Skip to content

Commit 06ff370

Browse files
sommiosommio
committed
docs(knowledge): add wake-trigger refresh integration solution
Document a pattern for canonicalizing ingest flow and durable success states in wake-triggered auto-refresh workflows.
1 parent feaec80 commit 06ff370

2 files changed

Lines changed: 370 additions & 0 deletions

File tree

‎docs/en/solutions/integration-issues/wake-triggered-feed-auto-refresh-must-reuse-canonical-ingestion-and-durable-success-state-2026-04-20.md‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
---
2+
title: Wake-triggered feed auto-refresh must reuse the canonical ingestion pipeline and durable success state
3+
date: 2026-04-20
4+
category: integration-issues
5+
module: feed auto refresh
6+
problem_type: integration_issue
7+
component: nest_service
8+
symptoms:
9+
- a same-process wake after sleep could leave feed data stale because `INGEST_ON_BOOT` only covered cold startup
10+
- the API had no durable global timestamp for the last successful wake-driven refresh, so elapsed-interval gating had nowhere authoritative to read from
11+
- any wake-refresh implementation that bypassed `FeedIngestionService` risked drifting away from the existing article-content and summary pipeline
12+
root_cause: missing_workflow_step
13+
resolution_type: code_fix
14+
severity: medium
15+
related_components:
16+
- background_job
17+
- typeorm_repository
18+
- typeorm_migration
19+
tags:
20+
[
21+
feed-auto-refresh,
22+
wake-resume,
23+
feed-ingestion,
24+
bootstrap,
25+
prisma,
26+
summary-pipeline,
27+
nestjs,
28+
]
29+
---
30+
31+
# Wake-triggered feed auto-refresh must reuse the canonical ingestion pipeline and durable success state
32+
33+
## Problem
34+
35+
`apps/api` could ingest feeds during bootstrap, but it still behaved like a startup-only path. When the same Node process resumed after machine sleep or runtime freeze, feeds could stay stale even though the app was alive again, and there was no durable global success timestamp to decide whether a background catch-up refresh was overdue.
36+
37+
The fix had to stay non-blocking for HTTP startup, preserve the existing meaning of `INGEST_ON_BOOT`, and keep newly discovered articles flowing into the existing article-content and LLM summary chain instead of inventing a second ingestion path.
38+
39+
## Symptoms
40+
41+
- Same-process wake/resume had no dedicated refresh trigger, so freshness depended on a later cold restart or manual intervention.
42+
- There was no single persisted `lastSuccessfulAutoRefreshAt` source of truth for interval checks; storing it on `Feed` rows would have turned a global decision into ambiguous per-feed state.
43+
- The existing ingestion path already owned persistence, Markdown enrichment, and summary scheduling, so a parallel wake-refresh loop would have been likely to diverge from the main contract immediately.
44+
45+
## What Didn't Work
46+
47+
- Reusing `INGEST_ON_BOOT` as the wake-refresh switch would have mixed two different semantics: unconditional bootstrap ingestion versus elapsed-interval refresh after same-process resume.
48+
- Putting wake-refresh eligibility on request paths was rejected because the requirements explicitly kept feed/network work out of read traffic.
49+
- Treating the last successful wake refresh as per-feed state did not fit the product rule. This slice needed one global timestamp, not a fan-out of duplicated values on `Feed`.
50+
51+
## Solution
52+
53+
The implementation landed three linked changes inside `apps/api`.
54+
55+
1. Add a dedicated durable singleton state model plus a defaulted interval config:
56+
57+
```prisma
58+
model FeedAutoRefreshState {
59+
id String @id
60+
lastSuccessfulAutoRefreshAt DateTime?
61+
createdAt DateTime @default(now())
62+
updatedAt DateTime @updatedAt
63+
}
64+
```
65+
66+
```ts
67+
const DEFAULT_FEED_AUTO_REFRESH_INTERVAL_HOURS = 6;
68+
69+
return {
70+
feedAutoRefreshIntervalHours:
71+
validated.FEED_AUTO_REFRESH_INTERVAL_HOURS ??
72+
DEFAULT_FEED_AUTO_REFRESH_INTERVAL_HOURS,
73+
// ...
74+
};
75+
```
76+
77+
2. Extend `FeedIngestionService` so wake-driven callers can reuse the canonical pipeline, pass trigger metadata, and receive a structured run result:
78+
79+
```ts
80+
async ingestFromOpml(
81+
opmlPath: string,
82+
options: FeedIngestionOptions = {},
83+
): Promise<FeedIngestionRunResult> {
84+
const trigger = options.trigger ?? "bootstrap";
85+
const maxAttemptsPerFeed = Math.max(1, options.maxAttemptsPerFeed ?? 1);
86+
// ...
87+
88+
return {
89+
failedCount,
90+
status: summaryStatus,
91+
successCount,
92+
totalFeeds: subscriptions.length,
93+
trigger,
94+
};
95+
}
96+
```
97+
98+
3. Add a wake-only orchestration layer that detects a same-process resume gap, skips duplicate in-flight runs, gates by the durable interval, and only advances the success timestamp after a run finishes with at least one successful feed:
99+
100+
```ts
101+
async checkHeartbeat(nowMs: number = Date.now()) {
102+
const gapMs = nowMs - this.lastHeartbeatAt;
103+
this.lastHeartbeatAt = nowMs;
104+
105+
if (gapMs < RESUME_GAP_THRESHOLD_MS) {
106+
return;
107+
}
108+
109+
await this.handleWakeResume(nowMs, gapMs);
110+
}
111+
112+
private async handleWakeResume(nowMs: number, gapMs: number) {
113+
if (this.isRefreshRunning) {
114+
return;
115+
}
116+
117+
const lastSuccessfulAutoRefreshAt =
118+
await this.repository.getLastSuccessfulAutoRefreshAt();
119+
120+
if (
121+
lastSuccessfulAutoRefreshAt &&
122+
nowMs - lastSuccessfulAutoRefreshAt.getTime() < intervalMs
123+
) {
124+
return;
125+
}
126+
127+
const result = await this.feedIngestionService.ingestFromOpml(
128+
config.feedOpmlPath,
129+
{
130+
maxAttemptsPerFeed: 3,
131+
trigger: "auto_refresh_resume",
132+
},
133+
);
134+
135+
if (result.successCount > 0) {
136+
await this.repository.markSuccessfulAutoRefresh(new Date(nowMs));
137+
}
138+
}
139+
```
140+
141+
Regression protection landed at the same time:
142+
143+
1. `apps/api/src/config/app-config.spec.ts` locks the default and validation rules for `FEED_AUTO_REFRESH_INTERVAL_HOURS`.
144+
2. `apps/api/src/feeds/feed-ingestion.service.spec.ts` locks structured run results plus bounded retry behavior.
145+
3. `apps/api/src/feeds/feed-auto-refresh.service.spec.ts` covers first-run, stale-state, fresh-state, in-flight, and full-failure timestamp semantics.
146+
4. `apps/api/e2e/feed-auto-refresh.e2e-spec.ts` proves that wake-triggered refresh still writes through the canonical article-content path.
147+
5. `apps/api/e2e/prisma-schema.e2e-spec.ts` proves the singleton table exists without polluting `Feed` rows.
148+
149+
## Why This Works
150+
151+
The key was to separate orchestration from ingestion ownership instead of forking the pipeline.
152+
153+
- `FeedAutoRefreshService` owns only wake detection, interval gating, and overlap prevention.
154+
- `FeedIngestionService` remains the single owner of feed fetch, persistence, Markdown enrichment, and summary scheduling.
155+
- `FeedAutoRefreshState` keeps the success timestamp global and durable, which matches the product rule and survives process restarts.
156+
- The returned run result lets the wake layer distinguish `all_success`, `partial_success`, and `full_failure` without reinterpreting ingestion logs.
157+
- Because the wake path calls the same ingestion backbone, new articles still enter the same downstream article-content and summary contracts as bootstrap-ingested articles.
158+
159+
## Prevention
160+
161+
- Keep bootstrap and wake semantics explicit. New callers should pass a `trigger` instead of overloading `INGEST_ON_BOOT`.
162+
- Preserve the singleton-state boundary in tests: schema coverage should keep proving that wake-refresh state lives outside `Feed`.
163+
- Keep both unit and e2e coverage around interval gating, in-flight dedupe, and canonical pipeline reuse; those are the fragile contracts of this slice.
164+
- When adding retry logic to ingestion callers, keep the retry budget and retryability decision close to `FeedIngestionService` so orchestration layers do not drift into custom fetch loops.
165+
- The final review handoff at `.claude/handoffs/2026-04-20-163932-ce-review-feed-auto-refresh.md` recorded follow-up concerns, but this branch intentionally shipped without additional code changes after that pass; future readers should treat it as review context, not as part of the implemented fix.
166+
167+
## Related Issues
168+
169+
- Moderate overlap: `docs/en/solutions/integration-issues/feed-ingestion-retries-missing-article-markdown-2026-04-17.md` + `docs/zh-Hans/solutions/integration-issues/feed-ingestion-retries-missing-article-markdown-2026-04-17.md` - same feed-ingestion recovery area and same canonical-pipeline principle, but a different root cause and a different trigger surface
170+
- `apps/api/src/feeds/feed-auto-refresh.service.ts`
171+
- `apps/api/src/feeds/feed-auto-refresh.repository.ts`
172+
- `apps/api/src/feeds/feed-ingestion.service.ts`
173+
- `apps/api/src/feeds/feed-ingestion.types.ts`
174+
- `apps/api/prisma/models/feed-auto-refresh-state.prisma`
175+
- `apps/api/prisma/migrations/202604200001_add_feed_auto_refresh_state/migration.sql`
176+
- `apps/api/e2e/feed-auto-refresh.e2e-spec.ts`
177+
- `apps/api/e2e/prisma-schema.e2e-spec.ts`
178+
- `apps/api/README.md`
179+
- `docs/en/brainstorms/2026-04-20-feed-auto-refresh-on-wake-requirements.md`
180+
- `docs/zh-Hans/brainstorms/2026-04-20-feed-auto-refresh-on-wake-requirements.md`
181+
- `docs/en/plans/2026-04-20-001-feat-feed-auto-refresh-on-wake-plan.md`
182+
- `docs/zh-Hans/plans/2026-04-20-001-feat-feed-auto-refresh-on-wake-plan.md`
183+
- `.claude/handoffs/2026-04-20-155612-feed-auto-refresh-on-wake.md`
184+
- `.claude/handoffs/2026-04-20-163932-ce-review-feed-auto-refresh.md`
185+
- GitHub issue search: no matching issues found via `gh issue list --search "feed auto refresh wake resume ingestion" --state all --limit 5`

‎docs/zh-Hans/solutions/integration-issues/wake-triggered-feed-auto-refresh-must-reuse-canonical-ingestion-and-durable-success-state-2026-04-20.md‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
---
2+
title: 唤醒触发的 feed 自动刷新必须复用 canonical ingestion 流水线与持久成功状态
3+
date: 2026-04-20
4+
category: integration-issues
5+
module: feed auto refresh
6+
problem_type: integration_issue
7+
component: nest_service
8+
symptoms:
9+
- 同一 Node 进程在休眠或 freeze 后恢复时，feed 数据可能继续陈旧，因为 `INGEST_ON_BOOT` 只覆盖冷启动
10+
- API 之前没有持久化的全局“上次唤醒刷新成功时间”，因此基于间隔的 gate 没有权威来源
11+
- 如果唤醒刷新绕开 `FeedIngestionService`，就很容易与现有 article-content 与 summary 流水线发生漂移
12+
root_cause: missing_workflow_step
13+
resolution_type: code_fix
14+
severity: medium
15+
related_components:
16+
- background_job
17+
- typeorm_repository
18+
- typeorm_migration
19+
tags:
20+
[
21+
feed-auto-refresh,
22+
wake-resume,
23+
feed-ingestion,
24+
bootstrap,
25+
prisma,
26+
summary-pipeline,
27+
nestjs,
28+
]
29+
---
30+
31+
# 唤醒触发的 feed 自动刷新必须复用 canonical ingestion 流水线与持久成功状态
32+
33+
## Problem
34+
35+
`apps/api` 之前只能在 bootstrap 阶段做 feed ingestion，所以它本质上仍是“启动时跑一次”的路径。同一个 Node 进程如果在机器休眠或 runtime freeze 之后恢复，feed 仍可能保持陈旧，而且系统也没有一个持久化的全局成功时间戳来判断“现在是否已经到了该补跑一次后台刷新”的时机。
36+
37+
这个修复必须同时满足几件事：不能阻塞 HTTP 可用性；不能改变 `INGEST_ON_BOOT` 现有含义；新发现的文章仍要继续流入既有的 article-content 与 LLM summary 链路，而不是再造一条独立的 ingestion 路径。
38+
39+
## Symptoms
40+
41+
- 同进程唤醒/恢复没有专门的刷新触发点，所以数据新鲜度仍然依赖下一次冷启动或人工干预。
42+
- 没有单一持久化的 `lastSuccessfulAutoRefreshAt` 作为间隔判断依据；如果把它塞到 `Feed` 行上，会把“全局决策”错误地变成含糊的“每个 feed 自己的状态”。
43+
- 现有 ingestion 路径已经拥有持久化、Markdown 富化和 summary 调度能力，因此再写一套平行的唤醒刷新循环，很快就会和主契约漂移。
44+
45+
## What Didn't Work
46+
47+
- 直接把 `INGEST_ON_BOOT` 当成唤醒刷新开关不行，因为它会混淆两个不同语义：无条件 bootstrap ingestion 与同进程恢复后的按间隔刷新。
48+
- 把唤醒刷新 eligibility 挂到请求路径上也不行，因为需求明确要求不要把 feed/network 工作耦合到读请求。
49+
- 把“上次唤醒刷新成功时间”设计成 per-feed 状态也不合适。这个切片需要的是一个全局时间戳，而不是把同一个概念复制到每条 `Feed` 记录上。
50+
51+
## Solution
52+
53+
这次实现把变更拆成了 `apps/api` 内部的三个联动部分。
54+
55+
1. 增加专用的持久化单例状态模型，以及带默认值的间隔配置：
56+
57+
```prisma
58+
model FeedAutoRefreshState {
59+
id String @id
60+
lastSuccessfulAutoRefreshAt DateTime?
61+
createdAt DateTime @default(now())
62+
updatedAt DateTime @updatedAt
63+
}
64+
```
65+
66+
```ts
67+
const DEFAULT_FEED_AUTO_REFRESH_INTERVAL_HOURS = 6;
68+
69+
return {
70+
feedAutoRefreshIntervalHours:
71+
validated.FEED_AUTO_REFRESH_INTERVAL_HOURS ??
72+
DEFAULT_FEED_AUTO_REFRESH_INTERVAL_HOURS,
73+
// ...
74+
};
75+
```
76+
77+
2. 扩展 `FeedIngestionService`，让唤醒调用方能够复用 canonical 流水线、传入触发来源，并拿到结构化运行结果：
78+
79+
```ts
80+
async ingestFromOpml(
81+
opmlPath: string,
82+
options: FeedIngestionOptions = {},
83+
): Promise<FeedIngestionRunResult> {
84+
const trigger = options.trigger ?? "bootstrap";
85+
const maxAttemptsPerFeed = Math.max(1, options.maxAttemptsPerFeed ?? 1);
86+
// ...
87+
88+
return {
89+
failedCount,
90+
status: summaryStatus,
91+
successCount,
92+
totalFeeds: subscriptions.length,
93+
trigger,
94+
};
95+
}
96+
```
97+
98+
3. 新增一个只负责唤醒场景的编排层：检测同进程恢复间隔、跳过已在运行中的重复触发、按持久化间隔 gate 执行，并且只有在整次运行结束且至少一个 feed 成功后才推进成功时间戳：
99+
100+
```ts
101+
async checkHeartbeat(nowMs: number = Date.now()) {
102+
const gapMs = nowMs - this.lastHeartbeatAt;
103+
this.lastHeartbeatAt = nowMs;
104+
105+
if (gapMs < RESUME_GAP_THRESHOLD_MS) {
106+
return;
107+
}
108+
109+
await this.handleWakeResume(nowMs, gapMs);
110+
}
111+
112+
private async handleWakeResume(nowMs: number, gapMs: number) {
113+
if (this.isRefreshRunning) {
114+
return;
115+
}
116+
117+
const lastSuccessfulAutoRefreshAt =
118+
await this.repository.getLastSuccessfulAutoRefreshAt();
119+
120+
if (
121+
lastSuccessfulAutoRefreshAt &&
122+
nowMs - lastSuccessfulAutoRefreshAt.getTime() < intervalMs
123+
) {
124+
return;
125+
}
126+
127+
const result = await this.feedIngestionService.ingestFromOpml(
128+
config.feedOpmlPath,
129+
{
130+
maxAttemptsPerFeed: 3,
131+
trigger: "auto_refresh_resume",
132+
},
133+
);
134+
135+
if (result.successCount > 0) {
136+
await this.repository.markSuccessfulAutoRefresh(new Date(nowMs));
137+
}
138+
}
139+
```
140+
141+
同时补了五层回归保护：
142+
143+
1. `apps/api/src/config/app-config.spec.ts` 锁住 `FEED_AUTO_REFRESH_INTERVAL_HOURS` 的默认值与校验规则。
144+
2. `apps/api/src/feeds/feed-ingestion.service.spec.ts` 锁住结构化运行结果与有界重试行为。
145+
3. `apps/api/src/feeds/feed-auto-refresh.service.spec.ts` 覆盖首次运行、过期状态、未过期间隔、运行中去重，以及 full-failure 时间戳语义。
146+
4. `apps/api/e2e/feed-auto-refresh.e2e-spec.ts` 证明唤醒触发的刷新仍然沿用 canonical article-content 路径。
147+
5. `apps/api/e2e/prisma-schema.e2e-spec.ts` 证明这个单例表存在，并且不会污染 `Feed` 行结构。
148+
149+
## Why This Works
150+
151+
关键点在于把“编排职责”和“ingestion 所有权”分开，而不是分叉整条流水线。
152+
153+
- `FeedAutoRefreshService` 只负责唤醒检测、间隔 gate 与并发去重。
154+
- `FeedIngestionService` 继续是 feed 抓取、持久化、Markdown 富化与 summary 调度的唯一 owner。
155+
- `FeedAutoRefreshState` 让成功时间戳保持全局且持久化，这和产品规则一致，也能跨进程重启保留状态。
156+
- 返回的 run result 让唤醒层可以区分 `all_success`、`partial_success` 和 `full_failure`，不用自己重新解释 ingestion 日志。
157+
- 因为唤醒路径调用的是同一条 ingestion 主干，所以新文章仍会进入与 bootstrap ingestion 相同的下游 article-content 与 summary 契约。
158+
159+
## Prevention
160+
161+
- 保持 bootstrap 与 wake 两套语义显式分离。以后新增调用方时，应传 `trigger`，不要继续复用 `INGEST_ON_BOOT` 来硬塞含义。
162+
- 在测试里保住“单例状态边界”：schema 覆盖应持续证明唤醒刷新状态存在于 `Feed` 之外。
163+
- 继续保留围绕 interval gating、运行中去重、canonical 流水线复用的单测和 e2e；这些是本切片最脆弱的契约。
164+
- 如果以后再给 ingestion 调用方增加 retry 逻辑，要把 retry budget 与 retryability 判断继续收拢在 `FeedIngestionService` 附近，避免编排层重新长出自定义抓取循环。
165+
- 最后一份评审交接 `.claude/handoffs/2026-04-20-163932-ce-review-feed-auto-refresh.md` 记录了后续关注点，但这个分支在那次评审之后并没有继续改代码；未来读者应把它当作评审上下文，而不是已落地修复的一部分。
166+
167+
## Related Issues
168+
169+
- 中度重叠：`docs/en/solutions/integration-issues/feed-ingestion-retries-missing-article-markdown-2026-04-17.md` + `docs/zh-Hans/solutions/integration-issues/feed-ingestion-retries-missing-article-markdown-2026-04-17.md` - 同属 feed-ingestion 恢复问题域，也都强调 canonical 流水线原则，但根因与触发面不同
170+
- `apps/api/src/feeds/feed-auto-refresh.service.ts`
171+
- `apps/api/src/feeds/feed-auto-refresh.repository.ts`
172+
- `apps/api/src/feeds/feed-ingestion.service.ts`
173+
- `apps/api/src/feeds/feed-ingestion.types.ts`
174+
- `apps/api/prisma/models/feed-auto-refresh-state.prisma`
175+
- `apps/api/prisma/migrations/202604200001_add_feed_auto_refresh_state/migration.sql`
176+
- `apps/api/e2e/feed-auto-refresh.e2e-spec.ts`
177+
- `apps/api/e2e/prisma-schema.e2e-spec.ts`
178+
- `apps/api/README.md`
179+
- `docs/en/brainstorms/2026-04-20-feed-auto-refresh-on-wake-requirements.md`
180+
- `docs/zh-Hans/brainstorms/2026-04-20-feed-auto-refresh-on-wake-requirements.md`
181+
- `docs/en/plans/2026-04-20-001-feat-feed-auto-refresh-on-wake-plan.md`
182+
- `docs/zh-Hans/plans/2026-04-20-001-feat-feed-auto-refresh-on-wake-plan.md`
183+
- `.claude/handoffs/2026-04-20-155612-feed-auto-refresh-on-wake.md`
184+
- `.claude/handoffs/2026-04-20-163932-ce-review-feed-auto-refresh.md`
185+
- GitHub issue 搜索：`gh issue list --search "feed auto refresh wake resume ingestion" --state all --limit 5` 未找到匹配 issue

0 commit comments

Comments
 (0)