Skip to content

README.md

Latest commit

 

History

History
265 lines (195 loc) · 5.93 KB

README.md

File metadata and controls

265 lines (195 loc) · 5.93 KB
Raw

Yiku Thumbnail Generator Service

A Dockerized Node.js service for generating video thumbnails using FFmpeg, with integration to Google Cloud Storage, Cloudflare R2, and Supabase.

Features

  • 🎬 Video Processing: Downloads videos from GCS and generates thumbnails using FFmpeg
  • ☁️ Cloud Storage: Uploads thumbnails to Cloudflare R2 with custom CDN URLs
  • 🗄️ Database Integration: Updates video records in Supabase with thumbnail URLs
  • 🐳 Dockerized: Easy deployment and management with Docker Compose
  • 🔒 Security: API key authentication and rate limiting
  • 🧹 Auto-cleanup: Automatic temporary file cleanup and disk usage monitoring
  • 🏥 Health Checks: Built-in health monitoring endpoint

Architecture

N8N Workflow → Thumbnail Service → GCS (download) → FFmpeg (process) → R2 (upload) → Supabase (update)

Prerequisites

  • Docker and Docker Compose
  • Google Cloud Service Account with GCS access
  • Cloudflare R2 credentials
  • Supabase project with service role key
  • FFmpeg and rclone (installed in container)

Environment Variables

Create a .env file in the project root:

# Service Configuration
PORT=3000
NODE_ENV=production
HOST=0.0.0.0

# Disk Management
MAX_DISK_USAGE=80
CLEANUP_INTERVAL=300000

# Temp Directories
TEMP_DIR=./temp/videos
THUMBNAIL_DIR=./temp/thumbnails

# Google Cloud Platform
GOOGLE_APPLICATION_CREDENTIALS=/app/credentials/yiku-gcloud-key.json
GCP_PROJECT_ID=your-gcp-project-id

# Cloudflare R2
R2_ACCOUNT_ID=your-r2-account-id
R2_ACCESS_KEY_ID=your-r2-access-key
R2_SECRET_ACCESS_KEY=your-r2-secret-key
R2_BUCKET_NAME=yiku-content
R2_PUBLIC_URL=https://cdn.yiku.studio

# Supabase
NEXT_PUBLIC_SUPABASE_URL=https://your-project.supabase.co
SUPABASE_SERVICE_ROLE_KEY=your-service-role-key

Setup

  1. Clone and navigate to the project:

    cd yiku-thumbnail

  2. Install dependencies:

    npm install

  3. Set up credentials:

    mkdir -p credentials
# Copy your GCP service account key to credentials/yiku-gcloud-key.json

  4. Configure rclone:

    # Ensure rclone is configured for R2 access
rclone config

Deployment

Using Docker Compose (Recommended)

# Build and start the service
npm run deploy

# Or manually:
docker-compose up -d --build

Using Docker Commands

# Build the image
npm run docker:build

# Run the container
npm run docker:up

Management Commands

# View service status
docker-compose ps

# View logs
npm run docker:logs

# Restart service
npm run docker:restart

# Stop service
npm run docker:down

# Test connections
npm run docker:test

API Endpoints

Health Check

GET /health

Returns service status, disk usage, and authentication status.

Generate Thumbnail

POST /generate
Content-Type: application/json

{
  "video_url": "gs://bucket/video.mp4",
  "video_id": "unique_video_id"
}

Update Video Thumbnail

POST /update-video-thumbnail
Content-Type: application/json

{
  "video_url": "gs://bucket/video.mp4",
  "thumbnail_url": "https://cdn.yiku.studio/thumbnails/thumb.jpg"
}

Integration with N8N

The service is designed to integrate with N8N workflows. After video generation is complete:

  1. Call the thumbnail service using an HTTP Request node
  2. Wait for completion (typically 3-5 seconds)
  3. Update the video status to "done" in your database

N8N HTTP Request Configuration

Method: POST
URL: http://yiku-thumbnail:3000/generate
Headers: Content-Type: application/json
Body: {
  "video_url": "{{ $json.video_url }}",
  "video_id": "{{ $json.video_id }}"
}

Network Configuration

The service runs on the root_default Docker network to communicate with N8N. Ensure both containers are on the same network:

# Check network configuration
docker network ls
docker network inspect root_default

Testing

Local Testing

# Test the service locally
curl http://localhost:3000/health

# Test thumbnail generation
curl -X POST http://localhost:3000/generate \
  -H "Content-Type: application/json" \
  -d '{"video_url": "gs://bucket/video.mp4", "video_id": "test123"}'

From N8N Container

# Copy test script to N8N container
docker cp test-n8n-connection.js n8n-n8n-1:/home/node/

# Run test from within N8N
docker exec -it n8n-n8n-1 node test-n8n-connection.js

Troubleshooting

Common Issues

  1. Connection Timeouts: Ensure both containers are on the same Docker network
  2. Authentication Errors: Verify credentials are properly mounted and accessible
  3. Disk Space: Check disk usage and cleanup settings
  4. FFmpeg Errors: Verify video format compatibility

Debug Commands

# Check container logs
docker-compose logs -f

# Check container status
docker-compose ps

# Check network connectivity
docker exec -it yiku-thumbnail ping n8n-n8n-1

# Check file permissions
docker exec -it yiku-thumbnail ls -la /app/credentials/

Security Considerations

  • API Keys: Use environment variables for sensitive credentials
  • Network Isolation: Service runs on internal Docker network
  • File Permissions: Credentials mounted as read-only
  • Rate Limiting: Built-in request rate limiting
  • Input Validation: Request validation and sanitization

Monitoring

  • Health Endpoint: /health for service status
  • Disk Usage: Automatic monitoring and cleanup
  • Logs: Structured logging with Winston
  • Metrics: Request timing and error tracking

Development

# Start development mode
npm run dev

# Run tests
npm test

# Build Docker image
npm run docker:build

License

MIT License - see LICENSE file for details.