Skip to content

README.md

Latest commit

 

History

History
200 lines (155 loc) · 6.23 KB

README.md

File metadata and controls

200 lines (155 loc) · 6.23 KB
Raw

🔠 TechScry

Python License Status

TechScry monitors AI/tech YouTube channels, summarizes videos using OpenAI, scores their relevance for each user, and delivers curated digests via email.

🚀 Features

  • 🔎 Video Discovery: Pulls from RSS feeds of followed and trending channels
  • 🗘️ Transcription & Summarization: OpenAI-based LLM summaries with chunk merging
  • 📊 Smart Scoring: Personalized GPT scoring based on per-user interests
  • 🗕 Curation Pool: Per-user prioritized digest queues with timestamps
  • 🧠 Summary Caching: Avoids redundant LLM calls via summary deduplication
  • 📧 Digest Delivery: Styled emails with cooldown-aware delivery logic
  • 🧪 Mocking & Previews: Dry runs, preview flags, HTML saves for dev & testing
  • 🧵 Multi-user Support: Fully isolated state per user (seen, skipped, scores)
  • 🔁 Script Looping: Background-ready with loop_runner.py
  • 📄 System Logging: Output to logs/pipeline_log.jsonl for visibility

🏠 Project Structure

techscry/
├── agents/                  # Email agent (SMTP)
│   └── email_agent.py
├── archive/                 # Deprecated modules
│   └── scorer.py
├── config/                  # Interest profile seed
│   └── interest_profile.json
├── control_plane/           # Orchestration logic
│   └── orchestrator.py
├── data/                    # Shared cache (summaries)
│   └── summary_cache.json
├── digests/                 # Saved email HTML output
├── docs/                    # Docs: DEVLOG, ROADMAP, DECISIONS
│   ├── DEVLOG.md
│   ├── ROADMAP.md
│   └── DECISIONS.md
├── frontend/                 # 📌 New: Next.js-based feed viewer
├── modules/                 # Core logic modules
│   ├── transcript_fetcher.py
│   ├── summarizer.py
│   ├── smart_scorer.py
│   ├── transcript_cache.py
│   ├── user_profile.py
│   ├── skip_cache.py
│   └── curation_pool.py
├── scripts/                 # CLI entrypoints (digest, pipeline)
│   ├── run_pipeline.py
│   ├── send_curated_digest.py
│   └── dev_send_digest.py
├── templates/               # Jinja HTML email templates
│   ├── digest_email.html
│   └── digest_email_safe.html
├── tests/mock/              # Preview/test fixtures
│   ├── mock_digest_data.json
│   └── mock_skipped_videos.json
├── users/                   # Per-user state & preferences
│   └── <user_id>/
│       ├── profile.json
│       ├── seen_videos.json
│       ├── skipped.json
│       └── digest_queue.json
├── utils/                   # Utilities (logger, cooldown, chunking)
│   ├── logger.py
│   ├── chunking.py
│   └── notification_gate.py
├── loop_runner.py           # Universal interval-based loop executor
├── .env.template            # Configuration template
└── requirements.txt

🔧 Setup

git clone https://github.com/YitzhakMizrahi/techscry.git
cd techscry
python -m venv .venv
source .venv/bin/activate  # or .venv\Scripts\activate on Windows
pip install -r requirements.txt
cp .env.template .env

Then edit .env with your own API and SMTP credentials.

👤 Add Users

Create a profile like this:

// users/<id>/profile.json
{
  "email": "user@example.com",
  "interests": {
    "keywords": ["gpt", "react", "openai"],
    "preferred_channels": ["Fireship"]
  },
  "notification_settings": {
    "notification_threshold": 0.6,
    "digest_threshold": 0.3,
    "max_per_digest": 5,
    "cooldown_hours": 12
  }
}

🧪 Development Flags

Flag Script Description
--dry-run run_pipeline.py Simulates pipeline, logs actions
--verbose run_pipeline.py Print full debug output
--preview send_curated_digest.py Open rendered HTML in browser
--log-only send_curated_digest.py Simulate sending, log only
--save-html send_curated_digest.py Save HTML digest to digests/
--email-safe Any sender Use email-compatible HTML

🔁 Background Automation

Use loop_runner.py with -m to schedule background tasks:

# ❗ Usage Reminder
All scripts must be run using `-m` from the project root. For example:

python -m loop_runner --script scripts.run_pipeline --interval 900 --args --dry-run
python -m loop_runner --script scripts.send_curated_digest --interval 900 --args --log-only

📃 Logs

Logs stored in logs/pipeline_log.jsonl track each user's:

  • Digest dispatch
  • Pipeline runs
  • Dry run/testing status

🌐 Frontend App (Next.js)

The frontend/ directory contains a fully client-rendered Next.js UI for digest preview and skipped video inspection.

⚡ Features

  • Digest cards with title, summary, relevance badge
  • YouTube modal player on click
  • Hover play icon for visual feedback
  • Skipped video listing
  • Responsive design (1–6 columns depending on screen size)

🧪 Dev Setup

cd frontend
pnpm install
pnpm dev

Visit http://localhost:3000/user/default for the default user’s feed.

🤩 Design Principles

  • Context-Aware Delivery: Respects user preferences, follows cooldowns
  • Minimal LLM Cost: Caching, chunking, and relevance filtering built-in
  • Scalable via JSON: Fully user-isolated—no DB or server required
  • Human-Friendly Previews: Digest HTML + mock data for local design testing

📚 Docs & Planning

Check the docs/ folder for:

Built with ❤️ by TechScry Labs