09-24-2021-jam.md

• Recursively Summarizing Books with Human Feedback, OpenAI.

• Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach, Tuo Zhao's group.

  • They study the setting of finetuning with weak supervision or without any labeled data
  • Issues: how to prevent overfitting to noise?
  • A contrastive self-learning framework: contrastive regularization and confidence-based weighting