Bidirectional Learning of Facial Action Units and Expressions via Structured Semantic Mapping across Heterogeneous Datasets
SSM presents a Structured Semantic Mapping (SSM) framework for bidirectional learning between Facial Action Units (AUs) and Facial Expressions (FEs) under heterogeneous datasets. Unlike prior one-way transfer (AU → FE), SSM enables mutual enhancement (AU ↔ FE) without requiring joint annotations, addressing inconsistencies in annotation granularity and data domains.
🚧 This paper is currently under review. Code will be released upon acceptance.

- Bidirectional Learning across Tasks
Establishes reciprocal knowledge transfer between fine-grained AUs and coarse-grained expressions. - Textual Semantic Prototypes (TSP)
Builds structured semantic anchors from textual descriptions with learnable prompts. - Dynamic Prior Mapping (DPM)
Learns a bidirectional, data-driven association matrix guided by FACS priors for cross-task alignment. - Heterogeneous Joint Learning
Enables training across datasets with different annotation formats (frame-level vs. clip-level).
• First systematic study of AU ↔ FE bidirectional learning under heterogeneous supervision
• Achieves state-of-the-art performance on multiple AU and DFER benchmarks
• Demonstrates that expression semantics can improve AU detection, not just the reverse
• Strong cross-dataset generalization and zero-shot transfer ability 
• AU datasets: BP4D, DISFA
• DFER datasets: DFEW, FERV39K, MAFW
SSM consistently outperforms single-task and baseline models across diverse dataset combinations. 
@article{li2026bidirectional,
title={Bidirectional Learning of Facial Action Units and Expressions via Structured Semantic Mapping across Heterogeneous Datasets},
author={Li, Jia and Zhang, Yu and Chen, Yin and Hu, Zhenzhen and Li, Yong and Hong, Richang and Shan, Shiguang and Wang, Meng},
journal={arXiv preprint arXiv:2604.10541},
year={2026}
}