Add model: BEATs pretrained audio models by isaac-chung · Pull Request #4352 · embeddings-benchmark/mteb

isaac-chung · 2026-04-05T20:15:40Z

Summary

Adds BEATs (Audio Pre-Training with Acoustic Tokenizers) pretrained model implementations
Three pretrained checkpoints: BEATs_iter1, BEATs_iter2, BEATs_iter3
Auto-downloads BEATs source files from the unilm repo since there is no pip package
Tested BEATs_iter2: 75% accuracy on ESC50, also ran on AudioSetMini 0.38 mAP (matches the paper)

Model checklist

I have filled out the ModelMeta object to the extent possible
I have ensured that my model can be loaded using:
- mteb.get_model(model_name, revision)
- mteb.get_model_meta(model_name, revision)
I have tested the implementation works on a representative set of tasks
The model is public, i.e., is available either as an API or the weights are publicly available to download

Notes

Checkpoints are hosted on OneDrive (not HuggingFace), so model_name must point to a local .pt file
BEATs source code (MIT license) is fetched at runtime to ~/.cache/torch/hub/beats_src/

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

KennethEnevoldsen

I would really love to get this on HF, even if it is not supported that way everything we have is on HF.

KennethEnevoldsen · 2026-04-05T21:40:59Z

+"""
+
+
+def _load_beats_modules():


Hmm don't we want to put it on huggingface instead?

I think something like this could work:

from huggingface_hub import hf_hub_download import torch with open(hf_hub_download("username/repo", "model.pt"), "rb") as f: obj = torch.load(f, map_location="cpu", weights_only=True)

KennethEnevoldsen · 2026-04-05T21:58:29Z

+logger = logging.getLogger(__name__)
+
+_BEATS_SRC_URL = "https://raw.githubusercontent.com/microsoft/unilm/master/beats"
+_BEATS_SRC_FILES = ["BEATs.py", "backbone.py", "modules.py"]


I would probably rather copy this in - (this could break very easily with change on their end)

Either than or put it together with the weight on hf

isaac-chung · 2026-04-05T22:07:23Z

So we didn't upload models like EvaCLIP to HF for the same reason - we don't maintain them. Could that be a caveat? That we can upload models but don't maintain them? Or we want to maintain them as well?

Samoed · 2026-04-06T05:39:29Z

+
+beats_iter1 = ModelMeta(
+    loader=BEATsWrapper,
+    name="microsoft/beats-iter1",


Is this pretrained or finetuned model? They have a lot of models and this is not clear which it is

I added the "Pre-trained Model" column

Are they then fine-utned on audioset?

Should we make sure they are eq. to the ones in the paper?

Should we rename to:
microsoft/beats-iter1-pretrained

Are they then fine-utned on audioset?

Yeah, I've added it as a training dataset in ModelMeta.

Should we make sure they are eq. to the ones in the paper?

we get the same scores as in the paper for AudioSetMini, but different for ESC50, maybe because we set it up differently? (cross validation)

Renamed models.

Hmm out classification tasks are different due sampling - do we have some of the other models in their table we can compare with to see if the ordering is approx. the same?

There's SpeechCommandsv2 too - we have the mini version.

Hmm looked to compare a few of the implementation I'm not sure exactly what to draw from these, but beats seems to be doing surprisingly poorly.

mdl ours theirs

beats 0.75 0.951

microsoft/msclap-2023 0.974

microsoft/msclap-2022 0.9095

CLAP (not sure which one) 0.967

MIT/ast-finetuned-audioset-10-10-0.4593 / AST 0.962 0.887

lyrebird/wav2clip / Wav2CLIP 0.721 0.860

We do seem to be maintaining relative ordering if CLAP == microsoft/msclap-2023.

Which dataset/task are these for?

… checkpoints to gitignore

github-actions · 2026-04-22T02:20:45Z

This pull request has been automatically marked as stale due to inactivity.

KennethEnevoldsen · 2026-04-25T11:33:53Z

@isaac-chung do we need to figure out what to do here?

isaac-chung · 2026-04-25T11:47:53Z

Re: model hosting, I guess we could a) copy the needed files, or b) host it in the mteb org on HF.

Re: correctness/repro scores from the paper, we have the mini/sampled versions of the datasets they used in the paper. We can see if their repo has more scores. Otherwise I'm not sure what else we can try.

KennethEnevoldsen · 2026-04-25T12:00:19Z

If we could move it all to a versioned repo, then I think that would be fine (e.g. hf)

KennethEnevoldsen · 2026-05-09T15:45:41Z

@isaac-chung this seems to have been getting stale - do we want to finish it up?

isaac-chung · 2026-05-09T16:12:08Z

Sure. But we might want to focus on the mveb effort first?

github-actions · 2026-05-24T02:23:09Z

This pull request has been automatically marked as stale due to inactivity.

github-actions · 2026-05-31T02:23:36Z

This pull request has been automatically closed due to inactivity.

isaac-chung and others added 2 commits April 5, 2026 23:10

Add BEATs pretrained model implementations

d066280

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Set revision for BEATs model metadata

86d5b09

isaac-chung force-pushed the beats branch from c9e2926 to 86d5b09 Compare April 5, 2026 20:42

KennethEnevoldsen reviewed Apr 5, 2026

View reviewed changes

Samoed reviewed Apr 6, 2026

View reviewed changes

Comment thread mteb/models/model_implementations/beats_models.py Outdated

Samoed reviewed Apr 6, 2026

View reviewed changes

isaac-chung added 2 commits April 6, 2026 10:19

Pin BEATs source to specific commit, use device for map_location, add…

5c880dd

… checkpoints to gitignore

Rename BEATs models to include pretrained suffix

065a26d

Samoed added the new model Questions related to adding a new model to the benchmark label Apr 7, 2026

github-actions Bot added the stale label Apr 22, 2026

github-actions Bot removed the stale label Apr 26, 2026

isaac-chung marked this pull request as draft April 27, 2026 15:03

KennethEnevoldsen mentioned this pull request May 9, 2026

model: Add LanguageBind video and audio model wrapper #4557

Open

github-actions Bot added the stale label May 24, 2026

github-actions Bot closed this May 31, 2026

Samoed deleted the beats branch June 9, 2026 16:32

mdl	ours	theirs
beats	0.75	0.951
microsoft/msclap-2023	0.974
microsoft/msclap-2022	0.9095
CLAP (not sure which one)		0.967
MIT/ast-finetuned-audioset-10-10-0.4593 / AST	0.962	0.887
lyrebird/wav2clip / Wav2CLIP	0.721	0.860

		"""


		def _load_beats_modules():

Uh oh!

Conversation

isaac-chung commented Apr 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Model checklist

Notes

Uh oh!

KennethEnevoldsen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

isaac-chung commented Apr 5, 2026

Uh oh!

Uh oh!

Samoed Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KennethEnevoldsen Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 22, 2026

Uh oh!

KennethEnevoldsen commented Apr 25, 2026

Uh oh!

isaac-chung commented Apr 25, 2026

Uh oh!

KennethEnevoldsen commented Apr 25, 2026

Uh oh!

KennethEnevoldsen commented May 9, 2026

Uh oh!

isaac-chung commented May 9, 2026

Uh oh!

github-actions Bot commented May 24, 2026

Uh oh!

github-actions Bot commented May 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

isaac-chung commented Apr 5, 2026 •

edited

Loading

Samoed Apr 6, 2026 •

edited

Loading

KennethEnevoldsen Apr 6, 2026 •

edited

Loading