readme.md

Multimodal Deception Detection

Can typing behavior and pretrained language representations reveal deception when the words alone are not enough?

An end-to-end machine-learning portfolio project combining narrative text, frozen Hugging Face transformer embeddings, and keystroke-derived behavioral features to identify the lie in a game of Two Truths and a Lie.

Built around the Motivated Deception Corpus, the pipeline covers multimodal data validation, grouped cross-validation, feature engineering, classical and transformer-based modeling, out-of-fold prediction generation, and score-level fusion.

Current status: The classical modeling, score-fusion, and frozen-transformer stages are complete. Natural extensions include transformer–behavior fusion, optional DeBERTa-v3 fine-tuning, and fully nested comparison of future model configurations.

The benchmark above provides context rather than a controlled head-to-head comparison. The original paper and this project use different training and evaluation pipelines.

What this project demonstrates

Capability	Implementation
Messy multimodal data processing	Converts nested pickle records into validated narrative-level data while parsing text, keystrokes, and timestamp sequences
Leakage-aware evaluation	Keeps each three-story triplet together across a fixed development/test split and grouped cross-validation folds
Feature engineering	Extracts typing duration, inter-key timing, pause, keystroke, backspace, and lexical features
Transformer transfer learning	Uses pretrained MiniLM and MPNet encoders from Hugging Face as frozen feature extractors for narrative text
Model comparison	Evaluates linear, probabilistic, tree-based, kernel, and neural classifiers on identical grouped folds
Multimodel fusion	Generates aligned out-of-fold probabilities and compares score averaging with learned stacking
Reproducible ML tooling	Moves reusable data, feature, split, and evaluation logic into src/ modules and saves reusable model outputs

Key findings

• Typing behavior carries stronger predictive signal than classical text features. The behavioral random forest reaches a development ROC AUC of 0.638, compared with 0.552 for the strongest classical text baseline.
• Frozen transformer representations improve on the classical text models. MPNet embeddings with an RBF SVM reach a ROC AUC of 0.577 and select the correct lie in 43.1% of development triplets.
• The transformer representation and downstream classifier both matter. MPNet outperforms MiniLM when paired with an RBF SVM, while linear classifiers perform substantially worse on both embedding spaces.
• More complex models are not automatically better. The RBF SVM outperforms the MLP on both frozen transformer representations, and averaging weak and strong transformer classifiers reduces performance.
• Classical text–behavior fusion remains the strongest current development approach. Equal averaging across five classical text models and five behavioral models selects the correct lie in 49.6% of development triplets.
• The best fusion method depends on the objective. Logistic stacking produces the strongest narrative-level discrimination, while equal averaging preserves the strongest within-triplet ranking.

System design

flowchart TB
    A[Motivated Deception Corpus] --> B[Validated narrative data]

    B --> C1[TF IDF text features]
    B --> C2[Typing behavior features]
    B --> C3[MiniLM embeddings]
    B --> C4[MPNet embeddings]

    C1 --> D1[Classical text models]
    C2 --> D2[Behavioral models]
    C3 --> D3[MiniLM classifiers]
    C4 --> D4[MPNet classifiers]

    D1 --> E1[Classical and behavioral OOF scores]
    D2 --> E1

    D3 --> E2[Transformer OOF scores]
    D4 --> E2

    E1 --> F1[Current averaging and stacking]
    E2 --> F2[Current transformer comparison]

    E1 -.-> F3[Planned transformer behavior fusion]
    E2 -.-> F3

    F1 --> G[Development evaluation]
    F2 --> G
    F3 --> G

Loading

Dataset and task

The Motivated Deception Corpus was collected through a competitive online game designed to motivate participants to write convincing lies.

The released corpus contains:

• 4,716 narratives
• 1,572 three-story triplets
• 1 lie and 2 truths per triplet
• narrative text
• raw keystroke sequences
• keystroke timestamps

The dataset is not redistributed in this repository.

Two related prediction tasks are evaluated:

1. Narrative classification: assign each story a probability of being deceptive.
2. Triplet lie selection: choose the highest-scoring story within each Two Truths and a Lie triplet.

Random performance for triplet selection is 33.3%. The original paper reported 35.7% human accuracy and 41.2% for its BERT-based three-alternative classifier.

Current development results

Method	Balanced Accuracy	Macro F1	ROC AUC	Average Precision	Triplet Accuracy
Classical text — logistic regression	0.537	0.537	0.552	0.378	0.385
Frozen MiniLM — RBF SVM	0.500	0.400	0.562	0.376	0.407
Frozen MPNet — RBF SVM	0.501	0.402	0.577	0.403	0.431
Behavioral random forest	0.589	0.591	0.638	0.470	0.473
Behavioral-model average	0.591	0.593	0.641	0.475	0.488
Classical text + behavior average	0.531	0.482	0.636	0.467	0.496
Logistic-regression stack	0.611	0.600	0.645	0.480	0.487

Results are pooled out-of-fold predictions from the development partition.

The frozen-transformer RBF models are most useful as ranking models. Their continuous scores provide above-chance ROC and triplet performance even though their default 0.5 classification thresholds remain dominated by the majority truth class.

Learned stacking results are exploratory. A fully nested evaluation would be required before treating them as final unbiased estimates.

Modeling approach

Classical text branch

• word-level TF-IDF unigrams and bigrams
• logistic regression
• Multinomial Naive Bayes
• random forest
• calibrated linear SVM
• TF-IDF → truncated SVD → MLP

Frozen transformer branch

Two pretrained Sentence Transformers are used without task-specific fine-tuning:

• sentence-transformers/all-MiniLM-L6-v2
  • 384-dimensional narrative embeddings
• sentence-transformers/all-mpnet-base-v2
  • 768-dimensional narrative embeddings

Each frozen representation is evaluated with:

• class-balanced logistic regression
• calibrated linear SVM
• RBF SVM
• MLP

The transformer weights remain unchanged. Only the lightweight downstream classifiers are trained using the deception labels.

Behavioral branch

Engineered typing features include:

• onset delay and active typing duration
• inter-key timing statistics
• pause counts at multiple thresholds
• total keystroke behavior
• backspace count and rate

The behavioral feature vectors are evaluated with:

• logistic regression
• random forest
• histogram gradient boosting
• RBF SVM
• MLP

Fusion branch

The current fusion analysis includes:

• classical text-model averaging
• behavioral-model averaging
• classical text + behavioral averaging
• logistic-regression stacking
• RBF-SVM stacking
• histogram-gradient-boosting stacking
• MLP stacking
• within-encoder transformer averaging
• cross-encoder MiniLM and MPNet averaging

Aligned transformer out-of-fold scores have also been saved for future fusion with the strongest behavioral models.

Evaluation discipline

The same split manifest and grouped development folds are reused across all models.

• three-story triplets remain intact across training and validation data;
• learned preprocessing is fit only on each training fold;
• downstream classifiers use identical grouped folds;
• frozen transformer encoders do not use deception labels;
• model comparison uses the same validation triplets;
• fusion uses out-of-fold base-model probabilities;
• both class-imbalance-aware and game-specific metrics are reported.

The initial classical text–behavior configuration was evaluated once on the original held-out test partition in 01_classical_ml.ipynb. Subsequent multimodel and frozen-transformer experiments are reported using grouped out-of-fold development predictions. Because the original test labels have already been examined, they are not used for further model selection.

The identifier supplied with the released records is treated as a triplet ID, rather than assumed to represent a unique human subject across game rounds.

Repository structure

multimodal-deception-detection/
├── README.md
├── LICENSE
├── environment.yml
├── figures/
│   ├── model_landscape.png
│   └── triplet_benchmark.png
├── notebooks/
│   ├── 00_dataset_profile.ipynb
│   ├── 01_classical_ml.ipynb
│   ├── 02_multimodel_fusion.ipynb
│   └── 03_frozen_transformer_features.ipynb
├── results/
│   ├── frozen_transformer_development_metrics.csv
│   └── minilm_development_metrics.csv
└── src/
    ├── data.py
    ├── evaluation.py
    ├── features.py
    └── splits.py

Large embedding matrices, model artifacts, and the original corpus are excluded from version control.

Project progress

• Validate and profile the multimodal corpus
• Engineer classical text and typing-behavior features
• Establish classical text and behavioral baselines
• Generate grouped out-of-fold probability vectors
• Compare averaging and learned score fusion
• Extract frozen MiniLM and MPNet embeddings with Hugging Face
• Evaluate linear and nonlinear classifiers on transformer features
• Save aligned transformer probabilities for later reuse

Possible extensions

The current repository represents a complete classical, multimodel-fusion, and frozen-transformer milestone. Natural extensions include:

• incorporating the strongest MPNet out-of-fold score into transformer–behavior late fusion;
• concatenating frozen MPNet embeddings with engineered typing features for feature-level fusion;
• fine-tuning a modern encoder such as microsoft/deberta-v3-base under fully nested grouped evaluation.

Reproducing the environment

conda env create -f environment.yml
conda activate deception-ml

Download Motivated_Deception_Corpus.pkl from the original dataset repository and store it locally. The source data are excluded from version control.

The first use of MiniLM or MPNet downloads the pretrained model files from Hugging Face and stores them in the local model cache.

Citation

@article{barsever2023motivated,
  author  = {Barsever, Dan and Steyvers, Mark and Neftci, Emre},
  title   = {Building and Benchmarking the Motivated Deception Corpus:
             Improving the Quality of Deceptive Text Through Gaming},
  journal = {Behavior Research Methods},
  volume  = {55},
  pages   = {4478--4488},
  year    = {2023},
  doi     = {10.3758/s13428-022-02028-7}
}

License and attribution

This is an independent portfolio project and is not affiliated with the original dataset authors. The dataset is not redistributed here. Original project code is licensed under the terms in LICENSE.