AIG-Audio-Image-Generation

This repository contains the implementation for a system designed to assist sound editors in navigating large sound libraries By leveraging a Variational Autoencoder (VAE) and the BEATs transformer architecture, the system generates representative visual "scene summaries" for audio files, reducing the reliance on ambiguous file names.

📝 Overview

Sound editing often involves tedious iterations through vast amounts of recorded audio This project provides a proof-of-concept focused on Urban City Square soundscapes. It extracts rich acoustic features and composes a dynamic visual scene by layering assets (e.g., people, cars, musicians) onto a static environmental template.

🛠️ Technical Implementation

1. Feature Extraction & Labeling

• Model: Utilizes BEATs (Bidirectional Encoder representation from Audio Transformers) to generate probability vectors.
• Ontology: Predictions are mapped to the Google AudioSet hierarchy
• Sliding Window Inference: Processes audio in 5-second windows ($W_L = 5s$) with a 1-second stride ($S = 1s$) to capture temporal evolution .

2. Probability Aggregation

To consolidate overlapping window predictions into a single representational probability, the system employs Power Mean Aggregation :

$$\overline{P}=\left(\frac{1}{N}\sum_{i=1}^{N}p_i^k\right)^{1/k}$$

Using $k=2$ or $k=3$ ensures that transient but significant events (like a car horn) are prioritized in the final image without being "diluted" by silence.

3. Image Composition Logic

• Z-Index Layering: Dynamic assets are superimposed following a specific layering order to maintain perspective (e.g., birds in the background, fountains in the middle-ground).
• Visual Scaling: Asset transparency is tied to the probability score; lower confidence results in higher transparency.

📊 Evaluation & Results

The effectiveness of the system was validated through a perceptual audio test with 17 participants.

• Accuracy: The system achieved an aggregated average accuracy of 69.1%.
• Ambiguity Analysis: Perceptual errors were analyzed using Multi-Dimensional Scaling (MDS) to map the latent space.
• Key Finding: The highest confusion occurred between "Band Motor" and "Marching Band," which was mathematically supported by their low Euclidean Distance (11.68) and high Cosine Similarity (0.7621).

🚀 Future Roadmap

• Environmental Scaling: Expanding to include "Forest" and "Coastal" templates.
• Scene Classification: Integrating DCASE 2025 Task 1 protocols to automate the identification of soundscapes and dynamically switch base templates.
• Human-in-the-Loop: Implementing feedback mechanisms for users to manually "correct" or swap icons, allowing the model to learn individual preferences.

📚 References

• [1] Schmid, F., et al. (2025). "Low-complexity acoustic scene classification... in the DCASE 2025 challenge."
• [2] Setlur, V., et al. (2005). "Semanticons: Visual metaphors as file icons."
• [3] Google Research. "AudioSet Ontology."

---

Developed as part of the Professional Option Final Project at Ecole Centrale de Nantes. *Author: William MBIONWU | Supervisor: Modan TAILLEUR