Skip to content

SETUP_GUIDE.md

Latest commit

 

History

History
186 lines (138 loc) · 5.72 KB

SETUP_GUIDE.md

File metadata and controls

186 lines (138 loc) · 5.72 KB
Raw

Arabic News Data Engineering Project - Setup Guide

📋 Prerequisites

  • Docker Desktop (v20.10+) with at least 8GB RAM allocated
  • Docker Compose (v2.0+)
  • Windows PowerShell or Command Prompt
  • Google Cloud Platform account with BigQuery enabled
  • API Keys:
    • RapidAPI key for Arabic News API
    • GROQ API key for Llama 3.1-8B

🔧 Configuration Steps

1. Environment Variables

The .env file contains all necessary configuration. Verify these values:

# API Keys (Already configured)
X_RAPID_API_KEY=82e0065f27mshb18dd7d4b586ee0p1bde38jsnaee89a9af5bd
GROQ_API_KEY=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

# GCP Configuration (Already configured)
GCP_PROJECT_ID=news-project-sa
GCP_KEY_FILE_Path=./kafka/config/keyfile.json

# Airflow Credentials (Default - Change for production)
_AIRFLOW_WWW_USER_USERNAME=airflow_admin
_AIRFLOW_WWW_USER_PASSWORD=airflow_admin

2. Google Cloud Setup

✅ Already Configured:

  • Service account key: kafka/config/keyfile.json
  • Project ID: news-project-sa
  • BigQuery connector configured in compose.yaml

Required Manual Step:

  1. Go to Google Cloud Console
  2. Navigate to BigQuery
  3. Create dataset: bq_kafka_dataset (if not exists)
  4. Ensure service account has BigQuery Data Editor permissions

3. File Structure Verification

✅ All Critical Files Present:

  • compose.yaml - Docker services configuration
  • .env - Environment variables
  • Makefile - Build commands
  • airflow/Dockerfile - Airflow custom image
  • airflow/dags/news_fetching_dag.py - Main DAG
  • etl/config/config.yml - ETL configuration
  • etl/spark-jobs/process.py - Spark processing job
  • kafka/config/keyfile.json - GCP credentials
  • kafka/config/bigquery-sink.json - BigQuery connector config
  • kafka/schemas/*.avsc - Avro schemas for 3 topics

🚀 Starting the Project

Using Docker Compose (Recommended for Windows)

cd C:\Users\MSI\Desktop\Arabic_News_Data_Engineering_project

# Start all services
docker-compose --profile all up --build -d

🔍 Verifying Services

Check Running Containers

docker ps --format "table {{.Names}}\t{{.Status}}"

Expected Services (28 total):

  • ✅ Airflow: webserver, scheduler, worker_1, worker_2, triggerer, redis, postgres
  • ✅ Kafka: kafka1, kafka2, zoo1, zoo2, schema-registry, connect, rest-proxy
  • ✅ Spark: master, worker_1, worker_2
  • ✅ Monitoring: prometheus, grafana, flower

🌐 Accessing Services

Service URL Credentials
Airflow UI http://localhost:8086 airflow_admin / airflow_admin
Spark Master http://localhost:8084 No auth
Spark Worker 1 http://localhost:8085 No auth
Spark Worker 2 http://localhost:8087 No auth
Prometheus http://localhost:9090 No auth
Grafana http://localhost:3001 Default: admin / admin
Flower (Celery) http://localhost:5555 No auth

🐛 Troubleshooting

Docker Desktop Issues

Problem: 500 Internal Server Error for API route

Solution Option 1 (Recommended - Faster):

  1. Right-click Docker Desktop tray icon → Pause
  2. Wait 5 seconds
  3. Right-click Docker Desktop tray icon → Resume
  4. Wait until tray icon shows "Docker Desktop is running"
  5. Retry the docker-compose command

Solution Option 2 (Full Restart):

  1. Right-click Docker Desktop tray icon → Quit Docker Desktop
  2. Wait 10 seconds
  3. Start Docker Desktop from Start menu
  4. Wait until tray icon shows "Docker Desktop is running"
  5. Retry the docker-compose command

💡 Tip: Pausing Docker Engine is faster and preserves container state, making it ideal for quick fixes without losing running containers.

Kafka Brokers Not Starting

Solution:

# Remove old containers
docker rm -f kafka1 kafka2 kafka-connect kafka-rest-proxy

# Restart Kafka services
docker-compose --profile all up -d kafka1 kafka2 kafka-connect kafka-rest-proxy

🧪 Testing the Pipeline

1. Trigger Airflow DAG

  1. Open Airflow UI: http://localhost:8086
  2. Login with airflow_admin / airflow_admin
  3. Find DAG: news_fetching_and_processing
  4. Click the play button to trigger manually

2. Verify BigQuery Data

  1. Go to BigQuery Console
  2. Navigate to news-project-sa → bq_kafka_dataset
  3. Check table ar_news_ana_topic
  4. Run query: 
    SELECT * FROM `news-project-sa.bq_kafka_dataset.ar_news_ana_topic` 
LIMIT 10

🛑 Stopping Services

# Stop all services
docker-compose --profile all down

# Stop and remove volumes (clean slate)
docker-compose --profile all down -v

💾 Resource Management

Pausing Docker Engine (Free Up Resources)

When you're not actively using the project but want to keep containers intact:

Pause Docker Engine:

  • Right-click Docker Desktop tray icon → Pause
  • This frees up CPU and memory while preserving all container state

Resume Docker Engine:

  • Right-click Docker Desktop tray icon → Resume
  • All containers resume from their paused state

💡 Benefits:

  • Instant pause/resume (no rebuild needed)
  • Preserves all container data and state
  • Frees up system resources immediately
  • Much faster than stopping and restarting services

When to Use Each Option

Action Use When Time Data Preserved
Pause Taking a break, freeing resources temporarily Instant ✅ Yes
Stop Services Done for the day, want to free all resources ~30s ✅ Yes (unless -v flag)
Full Restart Troubleshooting Docker issues ~2-3min ✅ Yes