Skip to content

Commit 5b93abd

Browse files
czytczyt
committed
Retain classified adult content without weakening safe search
Persist moderation decisions in an extensible categories table, classify static and LLM adult/spam findings instead of deleting them, and keep public search safe by default with explicit all/adult views. Carry classifications through incremental sync and migrate legacy moderation blocks so upgrades can recover previously suppressed hashes. Constraint: Default and existing API searches must not expose adult or spam records Constraint: Legacy copyright-style permanent blocks must remain enforced Rejected: Boolean columns on torrents | a separate taxonomy table preserves confidence, reason, review time, and future categories Rejected: Keep adult rows in blocked | blocked prevents rediscovery and cannot support explicit adult search Confidence: high Scope-risk: broad Directive: New public search paths must apply SearchFilter zero-value semantics; category sync needs its independent watermark Tested: gofmt check; go vet ./...; go test -race ./...; npm run lint; npm run build; bash -n scripts/sync-to-remote.sh; desktop/mobile visual QA score 94; independent code review Not-tested: Live LLM provider responses and remote SSH sync against a production database
1 parent 65b8376 commit 5b93abd

16 files changed

Lines changed: 751 additions & 209 deletions

File tree

‎README.md‎

Lines changed: 40 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
> Co-authored by [Kimi K3](https://www.kimi.com/).
66
7-
一个自建索引的磁力链接搜索网站：Go 后端通过 DHT 网络爬虫实时采集磁力链接元数据，自动过滤成人内容与垃圾信息，Next.js 前端提供干净的搜索界面。
7+
一个自建索引的磁力链接搜索网站：Go 后端通过 DHT 网络爬虫实时采集磁力链接元数据，分类并默认隐藏成人内容，过滤垃圾信息，Next.js 前端提供可控的搜索界面。
88

99
## 架构
1010

@@ -15,23 +15,26 @@ flowchart TB
1515
Scraper["Tracker scrape ranking<br/>BEP 15 batch scrape"]
1616
Fetcher["Metadata fetch<br/>BEP-9 workers"]
1717
Filter{"Filter engine<br/>keywords + heuristics + size"}
18-
Discard["Drop and count<br/>adult / spam / too small"]
18+
Adult["Mark category=adult"]
19+
Discard["Drop and count<br/>spam / too small"]
1920
DB[("SQLite")]
2021
Web["Next.js frontend"]
2122
Mod["LLM moderation<br/>OpenRouter free model"]
22-
Blocked[("blocked table")]
23+
Categories[("categories table")]
2324
2425
Crawler <-->|UDP| DHT
2526
Crawler -->|new infohashes| Scraper
2627
Scraper -->|ranked by seeders| Fetcher
2728
Fetcher --> Filter
28-
Filter -->|hit| Discard
29-
Filter -->|pass| DB
29+
Filter -->|adult| Adult
30+
Adult --> DB
31+
Filter -->|spam / too small| Discard
32+
Filter -->|general| DB
3033
Web <-->|REST API| DB
3134
DB -->|hourly incremental review| Mod
32-
Mod -->|adult/spam: delete| Blocked
35+
Mod -->|category + confidence + reason| Categories
3336
Mod -->|title ad-trim: clean_name| DB
34-
Blocked -.->|rejected on insert, no resurrection| DB
37+
Categories -.->|safe / all / adult views| Web
3538
```
3639

3740
- `server/` — Go 后端：DHT 爬虫（anacrolix/dht/v2）、BEP-9 元数据获取（anacrolix/torrent）、过滤引擎（中英文成人词表 + 垃圾启发式）、SQLite 存储（modernc.org/sqlite，无 cgo）、REST API（标准库 net/http）
@@ -121,7 +124,7 @@ LLM 审核相关（见下文「LLM 二次审核」）：
121124
| `MODERATION_INTERVAL` | `1h` | 审核周期 |
122125
| `MODERATION_BATCH_SIZE` | `100` | 每次请求送审的标题数 |
123126
| `MODERATION_MAX_BATCHES` | `20` | 每轮最多几批（0 = 不限），用于封顶开销 |
124-
| `MODERATION_DRY_RUN` | `false` | 只打日志不删除 |
127+
| `MODERATION_DRY_RUN` | `false` | 只打日志，不写入分类或审核时间 |
125128
| `MODERATION_TRIM_TITLES` | `true` | 顺带清理标题里的广告文字 |
126129

127130
### 前端
@@ -156,8 +159,10 @@ dhtsearch-server_<version>_<goos>_<goarch>.tar.gz
156159

157160
## API
158161

159-
- `GET /api/search?q=xx&page=1&page_size=20` — 搜索（q 为空返回最新收录），结果含拼好的 magnet 链接
160-
- `GET /api/stats` — 收录数、成人/垃圾/体积过滤计数、LLM 审核统计、爬虫状态
162+
- `GET /api/search?q=xx&page=1&page_size=20` — 安全搜索（默认排除 adult/spam，q 为空返回最新收录）
163+
- `GET /api/search?q=xx&include_adult=true` — 搜索 general + adult，仍排除 spam
164+
- `GET /api/search?q=xx&category=adult` — 只搜索成人分类；也支持 `general` / `spam` / `all`
165+
- `GET /api/stats` — 收录数、分类数、垃圾/体积过滤计数、LLM 审核统计、爬虫状态
161166
- `GET /api/healthz` — 健康检查
162167

163168
## DoS 防护
@@ -223,31 +228,46 @@ API 搜索路径加入同一防护规则；仅监听回环地址时则不需要
223228

224229
## 过滤策略
225230

226-
### 第一道：入库前的静态过滤
231+
### 第一道：入库前的静态分类与过滤
227232

228-
- **成人内容**：内置 230+ 中英文关键词（短词用词边界正则防误伤，如 Avatar/Avengers 不会被 "av" 误拦），对资源名和文件名匹配；JAV 番号等弱信号需叠加其他信号才判定
233+
- **成人内容**：内置 230+ 中英文关键词（短词用词边界正则防误伤，如 Avatar/Avengers 不会被 "av" 误拦），对资源名和文件名匹配；命中后以 `category='adult'` 入库，不再丢弃
229234
- **垃圾信息**：SEO 关键词堆叠、纯符号/超长名称、零大小或异常大小、超大文件数、随机字符串文件名占比等启发式规则
230235
- **体积下限**：总体积小于 `MIN_TORRENT_SIZE`（默认 100 MiB）的种子直接丢弃，滤掉假种、单图、纯链接/说明文件等垃圾
231236

232-
命中任一即丢弃并计入统计（`adult_filtered` / `spam_filtered` / `size_filtered`）。规则见 `server/internal/filter/`。
237+
垃圾或体积下限命中仍会丢弃并计入 `spam_filtered` / `size_filtered`；成人命中计入 `adult_classified` 并保留。规则见 `server/internal/filter/`。
233238

234239
### 第二道：LLM 二次审核（每小时）
235240

236-
静态词表只能拦住明写关键词的内容。后台每小时调用一次 OpenAI 兼容 API，对**尚未审核过**的库内记录做批量复审，判定为成人或垃圾的直接删除。
241+
静态词表只能识别明写关键词的内容。后台每小时调用一次 OpenAI 兼容 API，对**尚未审核过**的库内记录做批量复审。模型返回 `category`、`confidence` 和 `reason`，结果写入分类表，原始种子始终保留。
237242

238243
- **增量**：每行有 `reviewed_at` 标记，只为没审过的行付费，不会每小时重扫全库
239-
- **不会复活**：删除的 infohash 写入 `blocked` 表，爬虫与增量同步都不会再把它加回来
240-
- **失败安全**：API 报错时该批不标记已审，下轮重试；模型返回无法解析、标签未知或漏项一律按 `ok` 处理，绝不因不确定而删除
244+
- **默认安全**：普通搜索排除 `adult` 和 `spam`；成人内容必须显式选择“全部”或“成人”视图
245+
- **可重新分类**：规则变化时重置 `reviewed_at` 即可重审，无需重新爬取已经收录的元数据
246+
- **失败安全**：API 报错时该批不标记已审，下轮重试；模型返回无法解析、标签未知或漏项一律按 `ok` 处理
241247
- **开销可控**：`MODERATION_MAX_BATCHES × MODERATION_BATCH_SIZE` 即每小时上限（默认 2000 条／小时）
242248
- **提示词**：明确要求「盗版本身不算垃圾」，否则模型会把整个库都判成垃圾；见 `server/internal/moderator/moderator.go` 的 `systemPromptFor`
243249

244-
首次开启建议先跑一轮 dry-run 看日志，确认判定符合预期再放开删除：
250+
首次开启建议先跑一轮 dry-run 看日志，确认判定符合预期再写入分类：
245251

246252
```bash
247253
MODERATION_DRY_RUN=true go run ./cmd/server
248254
```
249255

250-
审核统计通过 `GET /api/stats` 的 `moderation` 字段暴露（`reviewed` / `adult_removed` / `spam_removed` / `errors` / `pending` / `blocked`）。
256+
审核统计通过 `GET /api/stats` 的 `moderation` 字段暴露（`reviewed` / `adult_classified` / `spam_classified` / `errors` / `pending` / `blocked`）。`blocked` 仅保留给版权等需要永久拒绝重新入库的场景。
257+
258+
分类信息与主体记录分表，便于以后增加 anime、movie、music 等类型：
259+
260+
```sql
261+
CREATE TABLE categories (
262+
info_hash TEXT PRIMARY KEY,
263+
category TEXT NOT NULL,
264+
confidence REAL NOT NULL DEFAULT 1,
265+
reason TEXT NOT NULL DEFAULT '',
266+
reviewed_at INTEGER NOT NULL DEFAULT 0
267+
);
268+
```
269+
270+
从旧版升级时，`blocked` 中历史 `adult` / `spam` 记录会一次性迁成分类占位并解除永久拒绝；由于旧版已经删除了主体元数据，只能等爬虫再次发现相同 infohash 后恢复，但恢复前后都不会泄漏到安全搜索。`copyright` 等永久移除原因不迁移。
251271

252272
### 顺带：标题去广告（`MODERATION_TRIM_TITLES`）
253273

@@ -360,7 +380,7 @@ echo 'DEPLOY_HOST=user@example.com' > .deploy.env # gitignored
360380
./scripts/sync-to-remote.sh # 手动同步一次
361381
```
362382

363-
原理：本地实例把数据写到 `local.db`，脚本按水位线（`last_sync_ts`）导出新增行到小 delta 文件，scp 到服务器后 `INSERT OR IGNORE` 合并（按 info_hash 去重），只传输增量。
383+
原理：本地实例把数据写到 `local.db`，脚本用独立水位线导出新增种子和后续分类变化到小 delta 文件，scp 到服务器后按 info_hash 合并，只传输增量。分类水位独立于种子的 `created_at`，因此已同步记录后来被 LLM 改成 adult/spam 时也会更新远端。
364384

365385
macOS 上可用 launchd 常驻/定时（安装后本地爬虫 API 在 `127.0.0.1:8089`，同步每 30 分钟一次）：
366386

@@ -374,7 +394,7 @@ macOS 上可用 launchd 常驻/定时（安装后本地爬虫 API 在 `127.0.0.1
374394

375395
| 路径 | 内容 |
376396
| --- | --- |
377-
| `~/Library/Application Support/dhtsearch/` | `bin/`、`local.db`、`last_sync_ts`、`.deploy.env`（可用 `DHTSEARCH_STATE_DIR` 覆盖） |
397+
| `~/Library/Application Support/dhtsearch/` | `bin/`、`local.db`、`last_sync_ts`、`last_sync_ts.categories`、`.deploy.env`（可用 `DHTSEARCH_STATE_DIR` 覆盖） |
378398
| `~/Library/Logs/dhtsearch/` | `crawler.log`、`sync.log` |
379399

380400
> 这不是洁癖：launchd agent 对非系统卷（`/Volumes/...`）没有 TCC 权限，仓库放在外置盘时 agent **读写都会失败**——spawn 直接以 `EX_CONFIG (78)` 退出，或每次文件操作报 `Operation not permitted`。`$HOME` 下的路径不需要任何授权。

‎scripts/sync-to-remote.sh‎

Lines changed: 26 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -33,12 +33,15 @@ DEPLOY_HOST="${DEPLOY_HOST:?DEPLOY_HOST is required (env or .deploy.env)}"
3333
DEPLOY_DIR="${DEPLOY_DIR:-/opt/dhtsearch}"
3434
LOCAL_DB="${LOCAL_DB:-$STATE_DIR/local.db}"
3535
WATERMARK_FILE="${WATERMARK_FILE:-$STATE_DIR/last_sync_ts}"
36+
CATEGORY_WATERMARK_FILE="${CATEGORY_WATERMARK_FILE:-${WATERMARK_FILE}.categories}"
3637
SQLITE=/usr/bin/sqlite3
3738

3839
[ -f "$LOCAL_DB" ] || { echo "local db not found: $LOCAL_DB"; exit 1; }
3940

4041
LAST=0
4142
[ -f "$WATERMARK_FILE" ] && LAST=$(cat "$WATERMARK_FILE")
43+
CATEGORY_LAST=0
44+
[ -f "$CATEGORY_WATERMARK_FILE" ] && CATEGORY_LAST=$(cat "$CATEGORY_WATERMARK_FILE")
4245

4346
TMP_DIR=$(mktemp -d)
4447
trap 'rm -rf "$TMP_DIR"' EXIT
@@ -49,23 +52,29 @@ DELTA="$TMP_DIR/delta.db"
4952
# during the export are picked up by the next run instead of being skipped.
5053
#
5154
# Columns are listed explicitly rather than SELECT *: the delta schema then
52-
# stays stable even when the two ends run different binaries, and reviewed_at
53-
# is deliberately left behind so the remote applies its own moderation policy
54-
# to synced rows.
55+
# stays stable even when the two ends run different binaries. Category rows are
56+
# copied separately so adult classifications cannot turn into general results
57+
# after a sync.
5558
"$SQLITE" "$LOCAL_DB" <<SQL
5659
PRAGMA busy_timeout=5000;
5760
ATTACH '$DELTA' AS d;
5861
CREATE TABLE d.torrents AS
5962
SELECT info_hash, name, total_size, file_count, files, created_at
6063
FROM torrents WHERE created_at > $LAST;
64+
CREATE TABLE d.categories AS
65+
SELECT c.info_hash, c.category, c.confidence, c.reason, c.reviewed_at
66+
FROM categories c JOIN torrents t ON t.info_hash = c.info_hash
67+
WHERE t.created_at > $LAST OR c.reviewed_at > $CATEGORY_LAST;
6168
SQL
6269

6370
COUNT=$("$SQLITE" "$DELTA" "SELECT COUNT(*) FROM torrents;")
64-
if [ "$COUNT" = "0" ]; then
71+
CATEGORY_COUNT=$("$SQLITE" "$DELTA" "SELECT COUNT(*) FROM categories;")
72+
if [ "$COUNT" = "0" ] && [ "$CATEGORY_COUNT" = "0" ]; then
6573
echo "nothing new to sync"
6674
exit 0
6775
fi
68-
NEW_MARK=$("$SQLITE" "$DELTA" "SELECT MAX(created_at) FROM torrents;")
76+
NEW_MARK=$("$SQLITE" "$DELTA" "SELECT COALESCE(MAX(created_at), $LAST) FROM torrents;")
77+
NEW_CATEGORY_MARK=$("$SQLITE" "$DELTA" "SELECT COALESCE(MAX(reviewed_at), $CATEGORY_LAST) FROM categories;")
6978

7079
REMOTE_TMP=$(ssh "$DEPLOY_HOST" "mktemp /tmp/dhtsearch-sync.XXXXXX.db")
7180
scp -q "$DELTA" "$DEPLOY_HOST:$REMOTE_TMP"
@@ -77,7 +86,18 @@ ATTACH '$REMOTE_TMP' AS d;
7786
INSERT OR IGNORE INTO torrents (info_hash, name, total_size, file_count, files, created_at)
7887
SELECT info_hash, name, total_size, file_count, files, created_at FROM d.torrents
7988
WHERE info_hash NOT IN (SELECT info_hash FROM blocked);
89+
INSERT INTO categories (info_hash, category, confidence, reason, reviewed_at)
90+
SELECT c.info_hash, c.category, c.confidence, c.reason, c.reviewed_at
91+
FROM d.categories c JOIN torrents t ON t.info_hash = c.info_hash
92+
WHERE 1
93+
ON CONFLICT(info_hash) DO UPDATE SET
94+
category = excluded.category,
95+
confidence = excluded.confidence,
96+
reason = excluded.reason,
97+
reviewed_at = excluded.reviewed_at
98+
WHERE excluded.reviewed_at >= categories.reviewed_at;
8099
\" && rm -f '$REMOTE_TMP'"
81100

82101
echo "$NEW_MARK" > "$WATERMARK_FILE"
83-
echo "synced $COUNT torrents (watermark: $LAST -> $NEW_MARK)"
102+
echo "$NEW_CATEGORY_MARK" > "$CATEGORY_WATERMARK_FILE"
103+
echo "synced $COUNT torrents and $CATEGORY_COUNT categories (watermarks: $LAST -> $NEW_MARK, $CATEGORY_LAST -> $NEW_CATEGORY_MARK)"

‎server/cmd/server/main.go‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -222,10 +222,6 @@ func main() {
222222
}
223223
fetcher.Run(ctx, feed, func(rec metadata.Record) {
224224
res := filter.Check(rec.Name, rec.Files, rec.TotalSize)
225-
if res.Adult {
226-
st.IncrStat("adult_filtered", 1)
227-
return
228-
}
229225
if res.Spam {
230226
st.IncrStat("spam_filtered", 1)
231227
return
@@ -234,17 +230,26 @@ func main() {
234230
st.IncrStat("size_filtered", 1)
235231
return
236232
}
237-
if err := st.Upsert(store.Torrent{
233+
torrent := store.Torrent{
238234
InfoHash: rec.InfoHash,
239235
Name: rec.Name,
240236
TotalSize: rec.TotalSize,
241237
FileCount: rec.FileCount,
242238
Files: rec.Files,
243239
CreatedAt: time.Now().Unix(),
244-
}); err != nil {
240+
}
241+
if res.Adult {
242+
torrent.Category = store.CategoryAdult
243+
torrent.CategoryConfidence = 1
244+
torrent.CategoryReason = strings.Join(res.Reasons, "; ")
245+
}
246+
if err := st.Upsert(torrent); err != nil {
245247
logger.Printf("store upsert: %v", err)
246248
return
247249
}
250+
if res.Adult {
251+
st.IncrStat("adult_classified", 1)
252+
}
248253
st.IncrStat("fetched", 1)
249254
})
250255
} else {

0 commit comments

Comments
 (0)