Avgust

This repository holds the source code and data for the paper "AVGUST: Automating Usage-Based Test Generation from Videos of App Executions". This paper was accepted in ESEC/FSE 2022 research track. You can find the accepted paper here.

AVGUST is a human-in-the-loop technique that assists developers in generating usage-based tests for mobile apps by learning from videos of app usages.

1. The main code base is under code folder. You can find the code related to the 4 main stages of Avgust:

Step Extraction: Processing app videos to generate app-specific UI events
IR Classification: Classifying UI screens and widgets
Model Generation: Generating app-independent usage models
Dynamic Test Generation: Dynamically generating usage-based tests for new apps

2. the KNN and MLP screen and widget classifiers are under KNNClassifier and MLP_Screen_Widget_Classification, respectively.
3. the definitions of canonical screens and canonical widgets are under IR folder
4. the intermediate embeddings output by our classifiers are under embeddings folder.
5. the IR Models, generated tests, and intermediate results (e.g., screenshots, cropped widgets, reverse engineered UI layout hierarchy) are under output folder.
6. the processed video frames, screenshots, cropped widgets, keyboard classifier's results of all the usages are under usage_data folder.

This repository also contains the software requirements and installation instructions for the provided artifact.