Skip to content

k2iser/IOT

BranchesTags

Repository files navigation

🚦 Traffic Monitoring System

Real-time event streaming service that connects Dahua IVSS cameras to Telefónica's IoT Platform (FIWARE).

Python 3.12+ License: MIT

📋 Table of Contents

🎯 Overview

This system provides real-time ingestion and processing of video analytics events from Dahua IVSS (Intelligent Video Surveillance System) cameras, forwarding them to Telefónica's IoT Agent JSON platform for integration with FIWARE Context Broker.

Event Types Supported

  • EVD (Electric Vehicle Detection / Traffic)

    • TrafficJunction - Vehicle crossing detection
    • TrafficFlowStat - Traffic flow statistics

  • CROWD (People Counting)

    • PeopleCounting - People counting events
    • Crowd / CrowdDensity - Crowd density monitoring
    • PassengerFlow - Passenger flow statistics
    • HumanTrait / SmartMotionHuman - Human detection
    • LineCounting / CrossLine - Line crossing detection

✨ Features

Core Functionality

  • ✅ Real-time Event Streaming - Continuous IVSS event stream processing
  • ✅ Event Deduplication - Configurable TTL-based deduplication
  • ✅ Privacy Protection - Automatic redaction of sensitive data (license plates)
  • ✅ Device Whitelisting - Allowlist-based device filtering
  • ✅ Name Mapping - Friendly names for devices

Reliability & Performance

  • ✅ Rate Limiting - Per-device token bucket rate limiter
  • ✅ Circuit Breaker - Automatic failure protection with cooldown
  • ✅ Automatic Reconnection - Resilient IVSS connection management
  • ✅ Hot Reload - Configuration reload without restart (SIGHUP)
  • ✅ Graceful Shutdown - Clean service termination

Monitoring & Observability

  • ✅ Web Dashboard - Modern, responsive real-time dashboard
  • ✅ Metrics Export - Comprehensive runtime metrics
  • ✅ Event Logging - NDJSON event log with rotation
  • ✅ Health Checks - Service and connectivity monitoring
  • ✅ Status Files - Runtime state persistence

🏗️ Architecture

┌─────────────────┐         ┌──────────────────┐         ┌─────────────────┐
│  Dahua IVSS     │ Stream  │  traffic.py      │  HTTPS  │  IoT Agent JSON │
│  Cameras        ├────────>│  (Event Pipeline)├────────>│  (FIWARE)       │
│  (10.0.0.12)    │  HTTP   │                  │  POST   │  (Telefónica)   │
└─────────────────┘         └──────────────────┘         └─────────────────┘
                                     │
                                     │ Runtime Files
                                     ▼
                            ┌─────────────────┐
                            │  /opt/traffic/  │
                            │  runtime/       │
                            │  - status.json  │
                            │  - events.json  │
                            │  - last_*.json  │
                            └─────────────────┘
                                     │
                                     │ HTTP API
                                     ▼
                            ┌─────────────────┐
                            │  dashboard.py   │
                            │  Web UI :5000   │
                            └─────────────────┘

Data Flow

  1. Ingestion - Connect to IVSS event stream (HTTP GET with streaming)
  2. Parsing - Extract JSON objects from stream
  3. Deduplication - Check event fingerprint against recent cache
  4. Privacy - Redact sensitive information (plates, etc.)
  5. Extraction - Parse event type and extract relevant fields
  6. Resolution - Map machine name → device serial → friendly name
  7. Filtering - Apply allowlist and rate limits
  8. Publishing - POST to IoT Agent JSON with circuit breaker protection
  9. Logging - Write to NDJSON log and update status files

📦 Installation

Prerequisites

  • Python 3.12 or higher
  • Linux system with systemd (recommended)
  • Network access to IVSS cameras and IoT Platform
  • VPN connection (if required for camera network)

Quick Start

# Clone repository
git clone https://github.com/yourusername/IOT.git
cd IOT

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install --upgrade pip
pip install -r requirements.txt

# Create configuration
cp .env.example .env
nano .env  # Edit with your settings

# Create runtime directories
sudo mkdir -p /opt/traffic/runtime
sudo mkdir -p /opt/traffic/config
sudo chown -R $USER:$USER /opt/traffic

# Create configuration files
touch /opt/traffic/allowlist.txt
echo '{}' > /opt/traffic/machine_to_serial.json
echo '{}' > /opt/traffic/camera_name_map.json

# Test traffic service
python traffic.py

# In another terminal, start dashboard
python dashboard.py

Production Installation

# Install as systemd services
sudo cp systemd/traffic.service /etc/systemd/system/
sudo cp systemd/dashboard.service /etc/systemd/system/

# Edit service files with your paths
sudo nano /etc/systemd/system/traffic.service
sudo nano /etc/systemd/system/dashboard.service

# Reload systemd and start services
sudo systemctl daemon-reload
sudo systemctl enable traffic dashboard
sudo systemctl start traffic dashboard

# Check status
sudo systemctl status traffic
sudo systemctl status dashboard

⚙️ Configuration

Environment Variables

See .env.example for all configuration options. Key settings:

IVSS Connection

TRAFFIC_IVSS_URL=http://10.0.0.12/cgi-bin/eventManager.cgi?...
CAMERA_USER=admin
CAMERA_PASS=your_password

IoT Agent

IOTA_JSON_BASE=https://iota-json.iotplatform.telefonica.com:18185/iot/json
IOTA_APIKEY_EVD=your_evd_apikey
IOTA_APIKEY_CRWD=your_crowd_apikey

Rate Limiting

IOTA_MAX_PER_MINUTE=60    # Max events per device per minute
IOTA_BURST=10             # Max events per second (burst)

Circuit Breaker

IOTA_CB_OPEN_AFTER=20     # Open after N consecutive failures
IOTA_CB_COOLDOWN_SEC=30   # Cooldown period in seconds

Configuration Files

allowlist.txt

Whitelist of allowed device serials (one per line):

ABC12345678
XYZ98765432
DEF11223344

Empty file = allow all devices.

machine_to_serial.json

Map machine names to device serials:

{
  "Camera-Front-Door": "ABC12345678",
  "Camera-Back-Gate": "XYZ98765432",
  "IPC-HFW5442E": "DEF11223344"
}

camera_name_map.json

Map device serials to friendly names:

{
  "ABC12345678": "Main Entrance",
  "XYZ98765432": "Rear Gate",
  "DEF11223344": "Parking Lot"
}

Hot Reload Configuration

Reload configuration files without restarting:

# Send SIGHUP signal to reload maps
sudo systemctl reload traffic

# Or manually
kill -HUP $(pgrep -f traffic.py)

🚀 Usage

Running Services

# Development mode
python traffic.py       # Terminal 1
python dashboard.py     # Terminal 2

# Production mode (systemd)
sudo systemctl start traffic
sudo systemctl start dashboard

# View logs
sudo journalctl -u traffic -f
sudo journalctl -u dashboard -f

Accessing Dashboard

Open browser: http://localhost:5000

The dashboard provides:

  • Real-time event metrics
  • Event rate charts (last 10 minutes)
  • System health indicators
  • Circuit breaker status
  • Last events (EVD/CROWD)
  • Debug information

Command Line Operations

# Check service status
sudo systemctl status traffic

# Restart service
sudo systemctl restart traffic

# Reload configuration
sudo systemctl reload traffic

# View recent logs
sudo journalctl -u traffic -n 100

# Follow logs
sudo journalctl -u traffic -f

# Check runtime status
cat /opt/traffic/runtime/status.json | jq .

📊 Monitoring

Health Check Endpoints

# Dashboard health check
curl http://localhost:5000/healthz

# Get system status
curl http://localhost:5000/api/status | jq .

# Get event rates
curl http://localhost:5000/api/rate | jq .

# Get last events
curl http://localhost:5000/api/last/evd | jq .
curl http://localhost:5000/api/last/crowd | jq .

Runtime Files

All runtime state is stored in /opt/traffic/runtime/:

  • status.json - Current system status
  • events.json - NDJSON log of all events (rotated at 10MB)
  • last_evd.json - Last vehicle/traffic event
  • last_crowd.json - Last people counting event
  • last_unknown.json - Last unrecognized event (debugging)

Metrics

Key metrics tracked:

  • total_events_received - Events received from IVSS
  • total_events_published - Events successfully published to IoTA
  • total_events_deduplicated - Duplicate events filtered
  • total_events_filtered - Events filtered by allowlist
  • total_rate_limited - Events delayed by rate limiter
  • consecutive_failures - Current failure streak
  • circuit_breaker.trip_count - Times circuit breaker opened

🔍 Troubleshooting

No Events Received

  1. Check IVSS connection:

    curl -u admin:password "http://10.0.0.12/cgi-bin/eventManager.cgi?action=attach&codes=%5BTrafficJunction%5D"

  2. Check VPN connectivity:

    ip link show ppp0
ping 10.0.0.12

  3. Verify credentials in .env

Events Not Published to IoTA

  1. Check circuit breaker status:

    curl http://localhost:5000/api/status | jq '.breaker'

  2. Check rate limiting:

    curl http://localhost:5000/api/status | jq '.metrics'

  3. Test IoTA connectivity:

    curl -X POST \
  "https://iota-json.iotplatform.telefonica.com:18185/iot/json?k=APIKEY&i=DEVICEID" \
  -H "Content-Type: application/json" \
  -d '{"test": "value"}'

No People Counting Data

  1. Check if events are being received:

    tail -f /opt/traffic/runtime/events.json | grep -i people

  2. Check last unknown event (might be using different field names):

    cat /opt/traffic/runtime/last_unknown.json | jq .

  3. Verify camera firmware supports people counting events

  4. Check IVSS URL includes people counting codes:

    codes=%5BPeopleCounting,Crowd,CrowdDensity%5D

VPN Keeps Disconnecting

This is likely a system-level issue with your VPN client. Check:

  1. VPN client logs:

    sudo journalctl -u pppd -f

  2. Network stability:

    ping -c 100 10.0.0.12

  3. Consider using a VPN keepalive script or auto-reconnect daemon

Circuit Breaker Stuck Open

# Check cooldown remaining
curl http://localhost:5000/api/status | jq '.breaker.cooldown_remaining'

# Force restart to reset
sudo systemctl restart traffic

📚 Documentation

📖 Complete Documentation Hub

For detailed documentation, troubleshooting, and development guides, see:

📁 docs/README.md - Main documentation hub

Quick links:

📚 API Reference

Dashboard REST API

GET /healthz

Health check endpoint.

Response: 200 OK

GET /api/status

Get complete system status.

Response:

{
  "service": "traffic",
  "version": "1.0.0",
  "server_time_utc": "2024-01-15T10:30:00Z",
  "ivss": {
    "connected": true,
    "last_event_utc": 1705318200
  },
  "iota": {
    "last_code": 200,
    "breaker_open": false
  },
  "families": {
    "EVD": {"total": 1234, "last_utc": 1705318200},
    "CROWD": {"total": 567, "last_utc": 1705318190}
  },
  "metrics": {
    "received": 2000,
    "published": 1801,
    "deduplicated": 150,
    "filtered": 49,
    "rate_limited": 10,
    "consec_fail": 0
  },
  "breaker": {
    "breaker_open": false,
    "consec_fail": 0,
    "trip_count": 2,
    "cooldown_remaining": 0
  },
  "maps": {
    "allowlist_count": 5,
    "machine_to_serial_count": 8,
    "name_map_count": 8
  },
  "dedup": {
    "queue_len": 150,
    "ttl_sec": 10
  },
  "infra": {
    "vpn_iface": "ppp0",
    "vpn_up": true,
    "ivss_host": "10.0.0.12",
    "ivss_ping": true
  }
}

GET /api/rate

Get event rates by family.

Response:

{
  "window_sec": 90,
  "EVD": {"rate": 0.1234, "count": 11},
  "CROWD": {"rate": 0.0567, "count": 5}
}

GET /api/last/evd

Get last vehicle/traffic event.

GET /api/last/crowd

Get last people counting event.

GET /api/last/unknown

Get last unrecognized event (debugging).

GET /api/events/tail?n=20

Get last N events from NDJSON log.

GET /api/events/recent?seconds=60

Get events within last N seconds.

GET /api/charts/timeline?minutes=10

Get timeline data for charting.

Response:

{
  "timestamps": [1705318140, 1705318200, ...],
  "evd": [5, 3, 7, ...],
  "crowd": [2, 4, 1, ...],
  "minutes": 10
}

🤝 Contributing

Contributions are welcome! Please:

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

Code Style

  • Follow PEP 8 guidelines
  • Use type hints (Python 3.12+)
  • Add docstrings to functions
  • Run black for formatting
  • Run flake8 for linting
  • Run mypy for type checking
# Format code
black traffic.py dashboard.py

# Lint
flake8 traffic.py dashboard.py

# Type check
mypy traffic.py dashboard.py

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

  • Dahua Technology for IVSS camera systems
  • Telefónica for IoT Platform (FIWARE)
  • FIWARE Foundation for Context Broker

📞 Support

For issues, questions, or contributions:

Built with ❤️ for smart city traffic monitoring

About

No description, website, or topics provided.

Resources

Readme

License

MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages