AI Weekly Insights Agent (Open-Source Project Overview)

This project automatically compiles high-value AI updates every week and outputs a readable, structured, and traceable Markdown weekly report. It can also run in OpenClaw Skill form.

---

Project Goals

This project is designed to solve three practical problems:

• Information is scattered: official announcements, industry media, papers, and open-source updates are spread across many platforms.
• Information is noisy: duplicate content is common, signal-to-noise ratio is low, and tracking consistency is difficult.
• Output is unstable: many tools can fetch data, but few can consistently produce readable weekly reports.

So this project focuses on stable weekly output, not on building a heavyweight news platform.

---

Current Features (v1)

• Default weekly time window: 7 days (168h)
• Automatic aggregation of: official announcements, industry news, open-source updates, research papers, and OpenClaw leaderboard
• Paper ratio control (default max: 20%)
• Source link and publish date preserved for each item
• OpenClaw leaderboard Top 3 output (rank, author, purpose, link)
• LLM-generated long-form Chinese interpretation (target: 200+ characters)
• Output path: daily_docs/ai_weekly_YYYYMMDD.md

---

Output Strategy

• News first: prioritize official and industry news
• Paper throttling: papers are capped at 20% by default
• Official backfill: widen the fetch window when official news is insufficient
• Traceability: publish date appears in both body and links section
• Visible errors: fetch failures are recorded under a dedicated "Fetch Errors" section

---

OpenClaw Skill Mode

Recommended triggers:

• Generate AI weekly report
• Generate this week's report
• Refresh OpenClaw leaderboard

Recommended command:

python3 run_daily_digest.py --use-llm --llm-provider auto --window-hours 168 --max-paper-ratio 0.2 --min-official-items 3

Output file:

• daily_docs/ai_weekly_YYYYMMDD.md

Security-related optional flags:

• --allow-insecure-ssl: allow SSL fallback on certificate errors (disabled by default)
• --allow-custom-llm-endpoint: allow non-allowlisted LLM host (disabled by default)
• LLM_ALLOWED_HOSTS: append trusted hosts via env (comma-separated)

---

Environment Variables and Key Security

This project supports two configuration approaches:

1. Local user .env
2. Environment variable injection from OpenClaw runtime

Required:

• ARK_API_KEY
• ARK_ENDPOINT_ID (recommended)

LLM supports two modes:

1. Ark mode (compatible with existing Volcengine setup)
2. OpenAI-compatible mode (for Chat Completions compatible gateways)

Ark mode variables:

• ARK_API_KEY
• ARK_MODEL
• ARK_ENDPOINT_ID (recommended for production)

OpenAI-compatible variables:

• OPENAI_API_KEY
• OPENAI_BASE_URL (if it ends with /v1, script auto-expands to /chat/completions)
• OPENAI_MODEL (optional, fallback applies if omitted)

Optional:

• DIGEST_WEBHOOK_URL

Open-source security conventions:

About OpenClaw runtime:

• If OpenClaw runtime injects variables, users can run directly.
• If not injected, users need local .env or manual export before running.
• Declare provider and env names clearly on the skill page to reduce scan mismatches.

External network access declaration (for security review):

• https://ark.cn-beijing.volces.com (Ark LLM API)
• endpoint from OPENAI_BASE_URL (OpenAI-compatible mode)
• https://topclawhubskills.com (OpenClaw leaderboard)
• RSS/news domains listed in sources.json
• optional notifications: Feishu/Lark or DingTalk webhook (only with --send)

Default security policy:

• all outbound URLs must be https://

• LLM endpoint host is allowlisted by default

• webhook host is limited to official Feishu/Lark/DingTalk domains

• relaxing these limits requires explicit flags

• Never commit a real .env

• Commit only .env.example

• .gitignore should exclude .env and generated report files

.env.example:

ARK_API_KEY=your_ark_api_key_here
ARK_ENDPOINT_ID=ep-your-endpoint-id
ARK_MODEL=Doubao-Seed-1.6-lite
OPENAI_API_KEY=
OPENAI_BASE_URL=
OPENAI_MODEL=
DIGEST_WEBHOOK_URL=

Real keys should exist only in runtime environments and must never appear in repositories, logs, screenshots, or issues.

---

Suggested Repository Structure

ai-weekly-agent/
  run_daily_digest.py
  sources.json
  SKILL.md
  README.md
  LICENSE
  .env.example
  .gitignore
  daily_docs/

---

Quick Start

python3 run_daily_digest.py --use-llm

Common parameters:

• --window-hours 168: weekly time window (default)
• --max-paper-ratio 0.2: upper bound for paper ratio
• --min-official-items 3: minimum target count for official news
• --focus-skill "tavily": track ranking for a specific OpenClaw skill

View Daily Metrics

Use the metrics script to quickly inspect delivery/failure/latency/source coverage:

python3 project/scripts/metrics_report.py

Common options:

• --date 2026-04-14: query a specific day
• --show-runs: include per-run details
• --json: output JSON for dashboards or downstream processing
• --runs-dir /opt/ai-news/app/runs: point to a custom runs directory

Source Categories

1. Official Announcements: OpenAI, Anthropic, Google AI, Meta AI, Microsoft AI official blogs
2. Research Papers: arXiv (cs.AI/cs.CL/cs.CV), Papers with Code, Semantic Scholar trending
3. Code & Open Source: GitHub Trending, Hugging Face Trending, Awesome repositories
4. Industry Media: TechCrunch AI, The Verge AI, VentureBeat AI, MIT Tech Review AI
5. Community Discussion: Reddit (r/MachineLearning, etc.), Hacker News, X (Twitter) lists
6. Chinese AI Media: Jiqizhixin, QbitAI, Xinzhiyuan, 36Kr AI columns
7. China Official Model Sources: official pages, developer platforms, and official social channels of domestic foundation model vendors

---

Phase-Based Implementation (Now Included)

Phase 1: Streamlit UI (Runnable First Version)

• Added app.py for interactive front-end.
• Keeps existing run_daily_digest.py core logic.
• UI supports parameter input, trigger run, markdown preview, and download.
• Users can input their own API keys in UI session memory.

Run:

pip install -r requirements.txt
streamlit run app.py

Phase 2: Agentization with LangChain + LangGraph

• Added agent_runtime.py: wraps current digest logic into reusable pipeline function.
• Added langgraph_agent.py: stateful graph with retry path (run -> retry -> persist -> end).
• Existing logic is reused directly (fetch, filter, summarize, render).

Phase 3: MCP + Observability + History

• Added mcp_bridge.py: MCP config discovery + run history writer.
• Added mcp_servers.example.json: example MCP server definitions (search/storage).
• Added run trace files under runs/<run_id>/trace.json.
• Added global history log runs/history.jsonl for recent runs.

To enable MCP, copy:

cp mcp_servers.example.json mcp_servers.json

Then update command/env based on your local MCP servers.

---

Public Web Mode (Visitors Bring Their Own Keys)

This project supports a public Streamlit page where each visitor enters their own API keys in the sidebar.

• Set PUBLIC_WEB_MODE=true on the server to force public-session behavior.
• Keys entered in sidebar are session-only (st.session_state) and are not written back to .env / .ENV or shared through process env overrides.
• Sensitive password fields are no longer prefilled from server environment variables in public mode.
• For third-party feeds, put nyt in PUBLIC_API_FEEDS and enter NYTIMES_API_KEY in UI (now grouped with GNews/Public API settings).
• If PUBLIC_API_FEEDS includes nyt but no NYT key is provided, UI will stop the run with a clear prompt.
• For public deployment, keep server .env minimal and avoid placing personal production keys in repository or host disk.

---

24/7 Deploy on Alibaba Cloud VM (Streamlit + Nginx)

Recommended production route:

1. Run Streamlit as a system service (systemd) on port 8501.
2. Put Nginx in front (ports 80/443) and reverse-proxy to 127.0.0.1:8501.
3. Enable HTTPS with Let's Encrypt.
4. Restrict cloud security group to 80/443 (and 22 only for admin IPs).

Detailed commands and sample configs are in:

• project/docs/aliyun-streamlit-deploy.md