Custom Newsletter Generator

Transform scattered web content into personalized, AI-curated newsletters tailored to your interests.

Features • Installation • Usage

Overview

This newsletter generator creates a personalized newsletter that highlights the most relevant articles, blog posts, and updates from your favorite sources! It uses LLMs to automatically analyze, rank, and summarize web content based on your specific interests and criteria.

Example of a generated newsletter:

Features

• Intelligent Relevance Estimation: Use state-of-the-art LLMs to rate the relevance of content based on your interests
• Summarization: Generate concise summaries of content, so you can quickly grasp the main points
• Automatic Text Extraction - Clean content extraction from nearly any webpage
• Slack & RSS Integrations: Automatically ingest new content via Slack and RSS feed integrations
• Newsletter Compilation: Create an easy-to-read newsletter with the most relevant content

Tech Stack

• LLM Integrations: OpenAI and Anthropic models
• Observability/Tracing: Langfuse
• Task Queue: Celery + Redis/Valkey for fast asynchronous processing
• Web Framework: FastAPI for a responsive web interface

Installation

Set up the Python environment

# optional: pyenv install 3.12.9
# optional: pyenv local 3.12.9
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Install Redis/Valkey

The application uses Redis as a message broker for Celery. To install Redis, run:

On Ubuntu/Debian-based systems:

sudo apt update
sudo apt install redis-server

If you're on Arch Linux, you can use Valkey:

sudo pacman -S valkey

Set up environment variables

Create a .env file and set the necessary variables. See the .env.example file for which values need to be set.

Usage

To start the application, you can either use the provided startup script (recommended) or start the services manually.

Starting the Application

Using the startup script (recommended)

You can use the provided startup script to set up the environment and start the services. This script requires tmux and xdg-utils to be installed. From the main directory, run:

./scripts/start_app.sh

the application should now start automatically. Note that this script was only tested on Ubuntu 24.04, it may not work on other distributions.

Manual Startup

If you prefer to start the application manually, follow these steps:

(Optional) Open an SSH tunnel for access to a remote database by opening a new terminal and running.

ssh [user]@[ssh_server] -L 5432:[remote_host]:[remote_port]

Then, in a new window, start Redis

sudo systemctl start redis-server

or Valkey if you're on Arch Linux

sudo systemctl start valkey

Start the Celery worker by opening a new terminal window and running

source venv/bin/activate
celery -A src.newsletter_generator.workers.celery:celery_app worker --loglevel=info

Finally, in a new terminal winow, start the webpage by running

source venv/bin/activate
uvicorn src.newsletter_generator.web.app:app --reload

To see the webpage, go to http://127.0.0.1:8000

Configuring your Preferences and Settings

To customize your newsletter and preferences, edit the config.yaml file. You can specify which criteria to use to determine whether content is (ir)relevant, as well as giving few-shot examples to guide the LLM in its decision-making process. You can also set your preferred LLM model, token budget, etc. in this file.