Datasets Processing: https://github.com/ioverho/emotion_datasets Datasets: https://surfdrive.surf.nl/files/index.php/s/gnDwUL3ziBTiwy3
Emotion recognition has been a goal of artificial intelligence research since the very beginning of our field. Despite this, emotion classification on text remains out of reach. One source of frustration is the many different theoretical frameworks for emotion. Available datasets for emotion classification similarly use various, very different labelling systems, with little overlap between datasets (see [1], [2] or Table 1 in the datasets repo).
As a result, it's very difficult to test the generalisation of emotion classifiers across different datasets---the label space of the target dataset is completely different from datasets seen during training.
Further reading:
- Bostan, L. A. M., & Klinger, R. (2018). An analysis of annotated corpora for emotion classification in text.
- Al Maruf, A., Khanam, F., Haque, M. M., Jiyad, Z. M., Mridha, M. F., & Aung, Z. (2024). Challenges and opportunities of text-based emotion detection: a survey. IEEE access, 12, 18416-18450.
In this project, instead of developing a new representation learner, we will be predicting the effectiveness of fine-tuning on an emotion dataset, before transferring to another downstream emotion dataset (i.e., intermediate dataset transfer). By doing so, we hope to discover structure within different datasets, and be able to predict which (types of) datasets serve as good pre-training corpora for other datasets.
OoD prediction techniques have recently seen much interest in cases where one wants to predict the performance of a model on a novel dataset, without having to evaluate the model on the entire dataset. This occurs when there is insufficient budget to train and evaluate the model, or when labels are not available.
One line of research uses gradient information to construct “Task Embeddings”. In this project, we will be using text and task embeddings to predict transfer performance between emotion labelling schemes.
Further Reading: 3. Achille, A., Lam, M., Tewari, R., Ravichandran, A., Maji, S., Fowlkes, C. C., ... & Perona, P. (2019). Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 6430-6439). 4. Vu, T., Wang, T., Munkhdalai, T., Sordoni, A., Trischler, A., Mattarella-Micke, A., ... & Iyyer, M. (2020). Exploring and predicting transferability across NLP tasks. arXiv preprint arXiv:2005.00770.
Let's assume an emotion classifier consists of a text-encoder,
Given some set of emotion datasets $\mathcal{D}{i}\in \frak{D}$, which comes with some utility function $\mu(\hat{y}, y)$ (a measure of quality of the model's predictions, e.g., accuracy, F1, MCC, R2), we want to measure and predict the loss in utility incurred when transferring between datasets: $$\mu(f(g(x;\theta{\mathcal{D}{j}});\phi{\mathcal{D}{i}}), y)-\mu(f(g(x;\theta{\mathcal{D}{i}});\phi{\mathcal{D}{i}}), y),~~i\not=j,(x,y)\sim \mathcal{D}{i}$$ In a perfect world, where all datasets are comprehensive and contain high-quality texts, and all models generalize perfectly, this difference would be 0. Unfortunately, this is not the case: models do overfit to their training corpora. However, if the difference is small, it means that $\mathcal{D}{j}$ is didactively useful for training models that generalize well to $\mathcal{D}{i}$, and potentially to other datasets that lie out-of-distribution (OoD).
As said, we want to be able to predict this difference, based on the properties of the different datasets and models. The field of OoD generalisation prediction aims to do exactly this. Specifically, given some new, unseen dataset,
$$\underset{\mathcal{D}{i}\in\mathfrak{D}}{\arg\max}~~\mu(f(g(x;\theta{\mathcal{D}{i}});\phi{\mathcal{D}{\text{unseen}}}), y),~~\mathcal{D}{\text{unseen}}\not\in \mathfrak{D},(x,y)\sim \mathcal{D}_{\text{unseen}}$$
The field of OoD generalisation prediction is big. Two early techniques, however, might be useful for our case. Achille et al. in [3] and Vu et al. in [4] used an approximate Fisher information matrix for the same model trained on different datasets to create a task representation space. The theory was that models that lie close to each other in this representation space would transfer to each other better.
I posit that a similar relationship holds in emotion datasets. If two datasets have similar emotions in them, then the transfer between two datasets should be better than before. I propose using the same technique (task representations) per emotion class to try to predict which dataset would serve as a good intermediate finetuning dataset for other emotion datasets. Specifically, I want us to build a regression model as, $$\mu(f(g(x;\theta_{\mathcal{D}{j}});\phi{\mathcal{D}{i}}), y)\sim \Delta(\mathtt{task_repr(\mathcal{D}{i})}, \mathtt{task_repr(\mathcal{D}{j})})+\Delta(\mathtt{text_repr(\mathcal{D}{i})}, \mathtt{text_repr(\mathcal{D}_{j})})$$
Get the datasets set up (see above links), and explore the different emotions that are present. Try to create a train/test split that excludes some datasets entirely (preferably ones that have a similar emotion labelling scheme in the training data still), and excludes some emotion classes from datasets that are still in the training data. The validation set should be a uniform sample of data from the training datasets.
Finetune text encoders (i.e. a modern BERT variant) on the different datasets in isolation. During this fine-tuning stage, the parameters of the encoder,
Transfer the fine-tuned text encoders to the different datasets. During this second stage fine-tuning, the encoder paramaters, $g(x;\theta_{\mathcal{D}j})$, should be kept frozen. These now encode information about the first fine-tuning dataset, $\mathcal{D}{j}$.
Once fine-tuned, evaluate the model on the new dataset to get our values for: $$\mu(f(g(x;\theta_{\mathcal{D}{j}});\phi{\mathcal{D}_{i}}), y)$$ For the prediction model, this will serve as the endogenous variable (i.e., the variable we'll be trying to predict).
Use the task and text representations method discussed in Achille et al. in [3] and Vu et al. in [4] to create class-conditional representations for each emotion in each dataset. These should be constructed using the same text encoder,
In practise, the task representations are constructed from the gradients of
Try to visualize these task representations using dimensionality reduction techniques (e.g., UMAP, T-SNE, PCA).
Compute the difference between different tasks using some distance (or similarity) function applied to different task and text embeddings. The differences between the representations of different tasks will serve as our exogenous variable (i.e., the variables we'll using for prediction).
Using this, we'll try to build a good regression model to predict the generalisability loss. The model will likely also need some additional variables (like mean dataset performance, dataset size). Feel free to suggest other interesting variables you think might correlate well with transferability.
- 19 May, 23:59, [Optional] Report draft (only for feedback, ungraded)
- 23 May, 13:00, Poster presentation session (10% of course grade)
- 26 May, 23:59, Research report (40% of course grade)
[Optional] You can write down discussion points for our meetings, or simply track your progress here.
Progress made this week:
- …
Discussion points:
- …
Goals for the next meeting:
- …
Progress made this week:
- …
Discussion points:
- …
Goals for the next meeting:
- …
Progress made this week:
- …
Discussion points:
- …
Goals for the next meeting:
- …
Progress made this week:
- …
Discussion points:
- …