Skip to content

README.md

Latest commit

 

History

History
191 lines (138 loc) · 11.6 KB

README.md

File metadata and controls

191 lines (138 loc) · 11.6 KB
Raw

Anomaly Detection Toolkit

This file contains anomaly detection related script/model/automation, and explanation.

Outline

  1. Univariate Anomaly Detection
  2. Multivariate Anomaly Detection
  3. Outlier Ensembles
  4. Novelty Detection vs Outlier Detection
    • Outlier Detection
    • Novelty Detection
  5. Multi-Models Distributed Computing
    • Python Environment
    • Distributed Computing of Spark
  6. ML Toolkit
image

This diagram is a quick decision map for choosing an anomaly-detection approach based on

  • (1) how many features you have
  • (2) whether you care about individual points or subsequences/shapes.

1) Univariate (single feature)

A. Extreme-Value (no time dependence)

When observations are independent and you only care about unusually large/small values. image

  • When to use: No strong temporal correlation or seasonality.
  • Typical methods: Robust z-score/quantiles, Kernel Density Estimation (KDE), Gaussian Mixture Models (GMM).
  • Output: Point anomalies (outliers by value).

https://medium.com/@injure21/kernel-density-estimation-for-anomaly-detection-715a945bc729
This article explain kernel density estimation for anomaly detection

https://github.com/GeneSUN/Anomaly_Detection_toolkit/blob/main/Model/KDE/FeaturewiseKDENoveltyDetector.py
This class defined the kernel density estimation model

https://colab.research.google.com/drive/1qC-Gry8py_Icl0V8zNedlIeX3HFEKHuY#scrollTo=WXniSVfCznS_
This notebook use real-world example for step-by-step explanation of the article.

B. Time-Series (with time dependence)

When the current value depends on recent history and/or there is seasonality. image

  • When to use: Autocorrelation, trends, clear daily/weekly patterns.
  • Typical methods: Exponential Smoothing / ETS, ARIMA / SARIMA, STL decomposition, residual-based thresholding.

https://medium.com/@injure21/time-series-anomaly-detection-with-arima-551a91d10fe4
https://github.com/GeneSUN/Anomaly_Detection_toolkit/blob/main/Model/ARIMA_anomaly/ARIMAAnomalyDetector.py
https://github.com/GeneSUN/Anomaly_Detection_toolkit/blob/main/Model/ARIMA_anomaly/ARIMAAnomalyDetectorFuture.py
https://github.com/GeneSUN/Anomaly_Detection_toolkit/blob/main/Model/ARIMA_anomaly/EWMAAnomalyDetector.py
https://colab.research.google.com/drive/1Gc7Em68p0ivqWJ98Cne7lyPb5TrTcZ-L#scrollTo=5CMO3pbLVvTt

C. Unusual Shape (subsequence anomalies)

When you care about segments that look abnormal, not just single points.
image

  • When to use: Pattern/shape deviations over a window (e.g., [xₜ,…,xₜ+W]).
  • Typical methods: Autoencoders (reconstruction error over windows), distance to shape prototypes, (optionally Matrix Profile/shapelet ideas).
  • Output: Subsequence anomalies (unusual patterns over time).

https://medium.com/@injure21/autoencoder-for-time-series-anomaly-detection-021d4b9c7909
https://github.com/GeneSUN/Anomaly_Detection_toolkit/blob/main/Model/AutoEncoder/AutoencoderAnomalyDetector.py
https://github.com/GeneSUN/Anomaly_Detection_toolkit/blob/main/Model/AutoEncoder/MultiTimeSeriesAutoencoder.py
https://colab.research.google.com/drive/174QBd3_2k3e88UyC__jLLG45Ukk-PMBx#scrollTo=DJ8JVhSGc70o

2) Multivariate (multiple features)

A. Proximity-Based (point anomalies in feature space)

When you care about the overall state across several features (e.g., 5G SNR, RSRP, RSRQ, …) rather than each feature separately.

  • When to use: Joint behavior across features matters (correlations, clusters).
  • Typical methods: K-Means (distance to centroid), k-NN distance, LOF, DBSCAN.
  • Output: Point anomalies in high-dimensional space.

https://medium.com/@injure21/types-of-anomalies-in-data-part-2-value-based-detection-5ad9fabb30a7
https://github.com/GeneSUN/Anomaly_Detection_toolkit/blob/main/Model/Proximity-based%20/KMeansOutlierDetector.py
https://github.com/GeneSUN/Anomaly_Detection_toolkit/blob/main/Model/Proximity-based%20/KNNOutlierDetector.py
https://github.com/GeneSUN/Anomaly_Detection_toolkit/blob/main/Model/Proximity-based%20/LOFOutlierDetector.py
https://github.com/GeneSUN/Anomaly_Detection_toolkit/tree/main/Model/Proximity-based%20/DBSCANOutlierDetector.py
https://colab.research.google.com/drive/1ot_fdYbEyg8WVg7n_fADoI69TOS9a5P8#scrollTo=KD3jJx5Rh5dx

B. Multivariate Time-Series (temporal + cross-feature)

Extend proximity ideas with time dependence across all features.

C. Multivariate Unusual Shape (subsequence anomalies across features)

Detect segments whose joint shape across features is unusual.

While extensions from (Proximity-Based ) A to multivariate time series (B) and multivariate unusual shape detection (C) provide richer modeling power, they also introduce higher complexity. As the models become more sophisticated, their interpretability tends to decrease.

3) Outlier Ensembles

There are two main ways to build ensembles for anomaly detection:

1. Independent (Parallel) Ensembles

  • Each detector runs separately on the same data.
  • Results are combined at the end (e.g., by score averaging, voting).
image

https://medium.com/@injure21/ensemble-methods-for-outlier-detection-79f9d9af4af0
https://medium.com/@injure21/ensemble-methods-for-outlier-detection-8b4572a66fe7
https://github.com/GeneSUN/Anomaly_Detection_toolkit/blob/main/Model/Proximity-based%20/EnsembleOutlierDetector.py
https://colab.research.google.com/drive/1ot_fdYbEyg8WVg7n_fADoI69TOS9a5P8#scrollTo=OpaEvwmVvr5z

2. Sequential Ensembles

  • Detectors are applied one after another.
  • Each stage refines or filters the results from the previous stage.

https://medium.com/@injure21/ensemble-methods-for-outlier-detection-2-sequential-ensemble-abff0fae80bc
https://colab.research.google.com/drive/1LqBRw-p1OCP7VJ0Qn4qAA3i2H58_UCky Screenshot 2025-09-22 at 10 50 44 AM

4) Novelty Detection vs Outlier Detection

  • Outlier Detection
    • Training and testing use the same dataset.
    • The data contains outliers, defined as observations that are far from the majority.
image
  • Novelty Detection
    • Training and testing are different.
    • The training data is clean (no outliers) and serves as a reference.
    • New incoming observations are evaluated against this reference to detect anomalies.
image

How to Use This Repository

This repository supports both Outlier Detection and Novelty Detection.
You can control the behavior with the parameter new_idx:

  • new_idx = "all" → Perform Outlier Detection (entire dataset used).
  • new_idx = slice(-1, None) → Perform Novelty Detection (detect only the last point).
  • new_idx = slice(-n, None) → Detect the last n points as novelties.

https://medium.com/@injure21/difference-between-outlier-detection-and-novelty-detection-f21c21ed0962 https://colab.research.google.com/drive/1Gc7Em68p0ivqWJ98Cne7lyPb5TrTcZ-L#scrollTo=UKrOIuztVvzw

5) Multi-Models Distributed Computing

In real-world applications, you often need to run many anomaly detection models in parallel.
For example, monitoring thousands of customers to detect irregular behavior in real time.

This can be achieved in:

  • Python (single-machine environment): Suitable for smaller datasets or prototyping.
  • Spark (distributed environment): Recommended for large-scale scenarios, enabling parallel anomaly detection across millions of records efficiently.
image

https://medium.com/@injure21/scaling-time-series-modeling-spark-multiprocessing-and-gpu-side-by-sid-e353445ae205 https://colab.research.google.com/drive/1OA3EKXqiuMsQ5loQM7MlJVl_OuBpWAAt#scrollTo=32fjpkeS-nYP

https://colab.research.google.com/drive/1qC-Gry8py_Icl0V8zNedlIeX3HFEKHuY#scrollTo=d82NG7mNxm0c https://colab.research.google.com/drive/1qC-Gry8py_Icl0V8zNedlIeX3HFEKHuY#scrollTo=RV6KSqCIP-U-

6) ML Toolkit

Preprocess

https://github.com/GeneSUN/Anomaly_Detection_toolkit/blob/main/Preprocess/TimeSeries_Preprocess.ipynb https://github.com/GeneSUN/Anomaly_Detection_toolkit/blob/main/Preprocess/TimeSeriesFeatureTransformerPandas.py

Feature Engineering and Hyperparameter

https://medium.com/@injure21/feature-engineering-for-unsupervised-outlier-detection-practical-strategies-and-pitfalls-30a1155e5853

Evaluation

https://medium.com/@injure21/outlier-detection-evaluation-4e58439f8299