Skip to content
BranchesTags

Repository files navigation

🎭 Real-Time Emotion Detection

Detect facial emotions live from your webcam — with a privacy-first design that never stores a single frame.

A clean, modular, end-to-end computer-vision pipeline built on Python, OpenCV, TensorFlow, and Keras. Detect faces, classify emotions in real time, and export privacy-safe session reports — all from one CLI.

CI Python License: MIT Code style: unittest Platform

📖 Table of Contents

🔍 Overview

Real-Time Emotion Detection turns any webcam into a live facial-emotion classifier. It ships as a single command-line application that covers the full machine-learning loop:

  • Probe available webcams and check local environment health.
  • Generate a tiny synthetic dataset for pipeline validation, or import the FER2013 dataset from Hugging Face.
  • Train a Keras CNN, evaluate a saved model, and predict the emotion in a still image.
  • Run real-time webcam inference with on-frame overlays.
  • Export privacy-safe JSON and HTML session reports.

The seven default emotion classes are: angry, disgust, fear, happy, sad, surprise, neutral.

Privacy by design: the application does not store webcam frames. Session reports contain aggregate metrics only (counts, timings, confidence, an emotion timeline) — never images.

💡 Why This Project

Most emotion-detection demos are a single throwaway script: a model file, a webcam loop, and no way to reproduce, evaluate, or trust the result. This project is the opposite — a small but complete, production-shaped reference for how to structure a real-time CV application:

  • Every stage — data, training, evaluation, inference, reporting — is a separate, testable module.
  • The pipeline is honest about model quality: it reads model metadata and warns on-screen when a starter or low-accuracy model is loaded.
  • It degrades gracefully — with no trained model present, it falls back to a clearly-labelled heuristic classifier so the app still runs.
  • It treats privacy as a feature, not an afterthought.

✨ Features

Feature Description
🎥 Real-time webcam inference Live face detection + emotion overlay with an on-screen FPS counter.
🧠 Trainable Keras CNN Augmented 4-block convolutional network (augmented_cnn_v2) with batch norm, dropout, and L2 regularization.
📊 Evaluation reports Evaluate any saved model against an image-folder dataset and export metrics to JSON.
🖼️ Still-image prediction Classify the emotion in a single photo, with optional full-image fallback.
📁 Dataset tooling Import FER2013 from Hugging Face or generate a tiny synthetic dataset for smoke tests.
📝 Privacy-safe session reports Export JSON + styled HTML summaries — metrics only, no frames.
🩺 Environment doctor One command reports Python, library versions, model status, and camera availability.
🔎 Camera probe Discover which webcam index actually works.
🪶 Graceful degradation Runs with a labelled heuristic classifier when no model is available.
🧪 Tested Unit tests across config, dataset, model, preprocessing, reporting, and data models.

🎬 Demo

📸 Screenshots & demo GIF coming soon. Drop a screenshot at docs/assets/screenshot.png and a screen recording at docs/assets/demo.gif, then uncomment the image tags above.

A run produces a live window with a green box around each detected face, an emotion label, a confidence score, the prediction source, and an FPS readout. When a report directory is supplied, it also writes a JSON + HTML summary like this:

reports/live-demo.json    # machine-readable metrics + emotion timeline
reports/live-demo.html    # styled, shareable session summary

🏗️ Architecture Overview

The webcam runtime is a thin coordinator over small, single-responsibility modules:

flowchart LR
    CAM[Webcam<br/>OpenCV VideoCapture] --> DET[HaarFaceDetector<br/>face_detection.py]
    DET --> PRE[preprocess_face<br/>48x48 grayscale]
    PRE --> CLF[EmotionClassifier<br/>Keras or Demo]
    CLF --> DRAW[Overlay + FPS<br/>camera.py]
    CLF --> TRK[SessionTracker<br/>session.py]
    TRK --> REP[JSON + HTML report<br/>reporting.py]
Loading

Flow: capture a frame → detect faces (Haar cascade) → crop and preprocess each face to a 48×48×1 grayscale batch → classify → draw overlays → aggregate privacy-safe metrics → export reports on exit.

A companion offline pipeline handles data and models:

flowchart LR
    HF[Hugging Face<br/>AutumnQiu/fer2013] --> DL[download-fer2013]
    DL --> IMG[Image folders<br/>train/ + validation/]
    IMG --> TR[train<br/>augmented CNN]
    TR --> MODEL[(emotion_model.keras<br/>+ metadata.json)]
    MODEL --> EV[evaluate]
    MODEL --> RUN[run / predict-image]
Loading

For a deeper walkthrough of modules, data flow, and design decisions, see docs/ARCHITECTURE.md.

📂 Repository Structure

realtime-emotion-detection/
├── src/emotion_detection/     # Application package
│   ├── app.py                 # CLI entrypoint (argparse subcommands)
│   ├── camera.py              # Webcam runtime & overlay drawing
│   ├── face_detection.py      # OpenCV Haar-cascade face detector
│   ├── preprocessing.py       # Face crop → model-ready batch
│   ├── emotion_model.py       # Keras & heuristic classifiers + factory
│   ├── training.py            # CNN architecture + training loop
│   ├── evaluation.py          # Saved-model evaluation
│   ├── inference.py           # Still-image prediction
│   ├── dataset.py             # FER2013 import + synthetic demo data
│   ├── reporting.py           # JSON + HTML session reports
│   ├── session.py             # Privacy-safe metric aggregation
│   ├── models.py              # Shared dataclasses (FaceBox, prediction)
│   ├── config.py              # RuntimeConfig + default labels
│   └── doctor.py              # Environment health checks
├── tests/                     # unittest suite
├── scripts/                   # Windows (.cmd) & Bash (.sh) helper wrappers
├── docs/                      # Architecture, roadmap, FAQ
├── assets/models/             # Trained model target (binaries git-ignored)
├── pyproject.toml             # Packaging + dependencies
└── requirements.txt           # Runtime dependencies

Note: data/, reports/, and trained model binaries (assets/models/*.keras) are intentionally git-ignored — they are large and reproducible. A fresh clone contains no model; see Quick Start to generate or train one.

🚀 Installation

Requirements: Python 3.11+ and a working webcam (for live detection).

# 1. Clone
git clone https://github.com/Philipcyrus/realtime-emotion-detection.git
cd realtime-emotion-detection

# 2. Create a virtual environment
python -m venv .venv

Activate it for your shell:

Shell / OS Command
macOS / Linux source .venv/bin/activate
Windows (PowerShell) .venv\Scripts\Activate.ps1
Windows (Git Bash) source .venv/Scripts/activate 
# 3. Install dependencies and the package (editable)
python -m pip install -r requirements.txt
python -m pip install -e .

This exposes both an emotion-detect console command and the python -m emotion_detection.app module entrypoint. The examples below use emotion-detect; the two are interchangeable.

⚡ Quick Start

1. Verify your environment

emotion-detect doctor

Reports Python and library versions, model status, and which cameras are available.

2. Find your camera

emotion-detect probe-camera --max-index 3

Use whichever index shows "opened": true.

3. Run live detection

emotion-detect run --camera-index 0

Press q or Esc to quit.

With no --model, the app runs the built-in demo classifier — a deterministic heuristic that keeps the demo usable but is not a trained emotion model (it is clearly labelled on-screen). To get real predictions, train a model first ↓

4. Train a real model (optional)

The offline pipeline downloads FER2013 from Hugging Face, trains, and evaluates:

# Import up to 4000 images per class
emotion-detect download-fer2013 --output data/fer2013_images --max-per-class 4000

# Train the augmented CNN
emotion-detect train \
  --train-dir data/fer2013_images/train \
  --validation-dir data/fer2013_images/validation \
  --model-out assets/models/emotion_model.keras \
  --epochs 60 --batch-size 64

# Run with the trained model + export a report
emotion-detect run --model assets/models/emotion_model.keras --report-dir reports

🛠️ Usage

The CLI is organized into subcommands. Run any with --help for full options.

emotion-detect <command> [options]
Command Purpose
run Start real-time webcam emotion detection (also the default with no subcommand).
doctor Check imports, model presence, and camera availability.
probe-camera Probe webcam indexes to find a working one.
download-fer2013 Download FER2013 from Hugging Face into image-folder format.
make-demo-data Create a tiny synthetic dataset for pipeline validation.
train Train a Keras emotion model from image folders.
evaluate Evaluate a saved model against a dataset.
predict-image Predict the emotion for a single still image.

Key run options

Flag Default Description
--camera-index 0 OpenCV webcam index.
--model (none → demo) Path to a Keras .keras model.
--labels 7 defaults Comma-separated label order matching model output.
--width / --height 1280 / 720 Requested capture resolution.
--report-dir (none) Directory to write JSON + HTML reports.
--session-name timestamp Name used for report files.
--headless off Run without a window (requires --max-frames).
--max-frames (none) Stop after N frames — useful for smoke tests / CI.

📎 Examples

Live detection with a report:

emotion-detect run \
  --camera-index 0 \
  --model assets/models/emotion_model.keras \
  --report-dir reports \
  --session-name live-demo

Headless smoke run (no window, 3 frames):

emotion-detect run \
  --model assets/models/emotion_model.keras \
  --headless --max-frames 3 \
  --report-dir reports --session-name smoke-webcam

End-to-end smoke pipeline (no real dataset needed):

# Generate tiny synthetic data
emotion-detect make-demo-data --output data/smoke_emotions --labels happy,neutral --samples-per-label 3

# Train a 1-epoch validation model
emotion-detect train \
  --train-dir data/smoke_emotions/train \
  --validation-dir data/smoke_emotions/validation \
  --model-out assets/models/smoke_emotion_model.keras \
  --metadata-out assets/models/smoke_emotion_model.metadata.json \
  --labels happy,neutral --epochs 1 --batch-size 2

# Evaluate it
emotion-detect evaluate \
  --model assets/models/smoke_emotion_model.keras \
  --data-dir data/smoke_emotions/validation \
  --metadata assets/models/smoke_emotion_model.metadata.json \
  --output reports/smoke_evaluation.json --batch-size 2

Predict a still image:

emotion-detect predict-image \
  --image path/to/face.png \
  --model assets/models/emotion_model.keras

Windows / Bash helper scripts: the scripts/ folder contains .cmd (Windows) and .sh (Bash) wrappers — e.g. scripts/train_model.cmd 60 64, bash scripts/smoke_pipeline.sh 0 — that call these same commands with sensible defaults.

⚙️ Configuration

Runtime behavior is controlled entirely through CLI flags (mapped to an immutable RuntimeConfig). Notable rules:

  • Labels default to the seven FER classes. When a model has a sibling *.metadata.json with a labels array, that order is used automatically — so trained models carry their own label mapping.
  • Headless mode requires --max-frames (so a windowless run always terminates).
  • Camera fallback: if the requested index is unavailable, run automatically tries indexes 0–3 before giving up.
  • Model warnings: metadata fields (note, low val_accuracy, starter_emotions training data) trigger an on-screen "labels not reliable" warning.

Preprocessing is fixed to a 48×48 grayscale batch ((1, 48, 48, 1)); pixel rescaling (1/255) is handled inside the model graph, so the training and inference paths stay consistent.

🎯 Design Philosophy

  • Small modules, single responsibilities. The webcam loop coordinates; it doesn't do detection, classification, or reporting itself.
  • Deferred heavy imports. cv2 and tensorflow are imported inside functions, so --help and unit tests stay fast and light.
  • Honesty over hype. The app surfaces model quality instead of hiding it, and the demo classifier is always labelled as heuristic.
  • Privacy first. No frame ever leaves memory or hits disk; reports are metrics-only.
  • Reproducible, not magical. Data and models are regenerable from documented commands rather than committed binaries.

📈 Performance Notes

  • Face detection uses OpenCV's bundled Haar frontal-face cascade — CPU-only, no GPU required, fast enough for real-time on typical laptops. It favors frontal faces and good lighting.
  • Inference runs on CPU by default; a supported GPU/TensorFlow build accelerates training substantially but is not required to run.
  • Model accuracy: the augmented_cnn_v2 architecture reaches roughly 55–56% validation accuracy on 7-class FER2013 (~32k images). That is a solid baseline for this notoriously noisy dataset — FER2013 human accuracy is around 65% — but it is not a production-grade emotion recognizer. Expect reasonable but imperfect webcam labels.
  • FPS is displayed live and recorded in session reports; it depends on resolution, CPU, and the number of faces per frame.

❓ FAQ

The app shows my face but the emotion label is wrong. Is it broken?

No — that means the camera and detection paths work. Emotion accuracy is a model-quality matter. If you're on the demo classifier or a low-accuracy model, train on real FER2013 data (see Quick Start).

Why is there no model file after cloning?

Model binaries are git-ignored (they're large and reproducible). Train one, or the app falls back to the labelled demo classifier.

Does it upload or store my webcam video?

No. Frames stay in memory and are never written to disk. Reports contain only aggregate metrics.

More questions are answered in docs/FAQ.md.

🗺️ Roadmap

Planned directions include DNN-based face detection, additional dataset importers, and packaging improvements. See docs/ROADMAP.md for current / near-term / future / out-of-scope items.

🤝 Contributing

Contributions are welcome! Please read CONTRIBUTING.md for setup, tests, coding conventions, and the PR/issue workflow. By participating you agree to our Code of Conduct.

Run the test suite before opening a PR:

python -m unittest discover -s tests

📄 License

Released under the MIT License. See LICENSE.

🙏 Acknowledgements

About

Real-time webcam emotion detection app built with Python, OpenCV, TensorFlow, and Keras.

Resources

Readme
MIT license

Code of conduct

Code of conduct

Contributing

Contributing

Security policy

Security policy
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages