dataset: Add INatSounds

[isaac-chung]

New Dataset: INatSounds

Adds the iNaturalist Sounds dataset for audio-based species classification.

Dataset details

• HuggingFace: https://huggingface.co/datasets/mteb/inat_sounds
• Paper: https://arxiv.org/abs/2506.00343
• Task type: AudioClassification
• Modality: Audio
• License: CC-BY-4.0

The dataset contains recordings of 5,500+ species across birds, insects, amphibians, mammals, and reptiles, contributed by 27,000+ citizen scientists. The test split contains 49,527 recordings from 2023 observations.

Checklist

• Explain how this dataset fills a gap in MTEB
  • First large-scale bioacoustics species classification benchmark in MTEB
• Verify the dataset runs with the mteb package
• Test with sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
• Test with intfloat/multilingual-e5-small
• Confirm performance isn't trivial or random
• Complete all TaskMetadata fields
• Place task in mteb/tasks directory
• Import in __init__.py

[KennethEnevoldsen]

would be great to run at least one model on it to make sure that everything works as intended, but otherwise it looks good

It also seems like we are missing the descriptive stats

[isaac-chung]

I ran mini on Yamnet and for like 0.01 as main score - maybe species level is quite hard for it. Wonder how other models perform on this.

[KennethEnevoldsen]

hmm, yeah that seems very low - we talked about also implementing a domain specific model - might be better to test with one of those

[isaac-chung]

Yeah might look into one of these: OpenBEATs - a shikhar7ssu Collection https://share.google/nJLukTgjGY8mWEgZD

[KennethEnevoldsen]

We only have one label pr class? I think this is due to the downsample - I don't think that makes a lot of sense there

[isaac-chung]

Likely due to down sample yeah. Full set is 49k rows, and needs 30+GB to load

[KennethEnevoldsen]

We could still make it smaller, but probably good to keep it large enough to be meaningful.

How about:
a stratified subsample of the test set from 49,527 to 2 of each unique label (so about 10k) and then ensure that we have 10 labels per unique label so around 50-60k?

I believe currently it would sample the entire dataset and reembed it every time during evaluation. This could be optimized (e.g. by sampling all the IDs, then encoding then encoding only the required documents once), but it is a problem in the task not the dataset. In this case to improve runtime we could reduce the number of experiments to 1. I suspect with the current setting all 10 experiment are essentially the same anyway.

[isaac-chung]

Also the current label is at the most granular species level. Maybe we want to explore something higher level?

  ┌───────────────┬──────────┬────────────────┐
  │     Level     │ Distinct │     Range      │
  ├───────────────┼──────────┼────────────────┤
  │ supercategory │ 5        │ too few        │
  ├───────────────┼──────────┼────────────────┤
  │ order         │ 62       │ good candidate? │
  ├───────────────┼──────────┼────────────────┤
  │ family        │ 356      │ good candidate? │
  ├───────────────┼──────────┼────────────────┤
  │ genus         │ 2,075    │ over 2k        │
  ├───────────────┼──────────┼────────────────┤
  │ species       │ 5,569    │ over 2k        │
  └───────────────┴──────────┴────────────────┘

[KennethEnevoldsen]

This could be optimized (e.g. by sampling all the IDs, then encoding then encoding only the required documents once)

I am actually not entirely sure on this part seems that there is a cache it place that re-uses embeddings. So I don't think there is an issue. On that point, but still believe that the due to the large number of unique labels we end up sampling the whole dataset (we can check this if you have the results file)

[KennethEnevoldsen]

Also the current label is at the most granular species level. Maybe we want to explore something higher level?

Ahh yeah that could be a way out. Feels like hiearchical classification. I would probably suggest family, genus, species but let me ask a friend who is a biologist about what makes sense here.

I am probably leaning toward species still though

[KennethEnevoldsen]

talked with biology friend, she argued that it makes no sense to predict up the their - species or maybe genus.

She however suggest downsampling the number of species to e.g. 2k - I agree that this is probably the best approach. We could also split it into 5 datasets one for each subcategory (I don't know what a subcategory is though)

[isaac-chung]

Makes sense to down sample species. Thanks for asking!

[KennethEnevoldsen]

No problem - I also heard that a large majority of the dataset is birds (I think ~4k)

[github-actions]

This pull request has been automatically marked as stale due to inactivity.

[isaac-chung]

Going to stick with species and run the random encoder

[isaac-chung]

random encoder:

{
  "dataset_revision": "6db625bf80d71ded64690022f0df56ceae972b6e",
  "task_name": "INatSounds",
  "mteb_version": "2.12.21",
  "scores": {
    "test": [
      {
        "scores_per_experiment": [
          {
            "accuracy": 0.000848,
            "f1": 0.000499,
            "f1_weighted": 0.000586,
            "precision": 0.000402,
            "precision_weighted": 0.00063,
            "recall": 0.001208,
            "recall_weighted": 0.000848,
            "ap": null,
            "ap_weighted": null
          },
          {
            "accuracy": 0.000969,
            "f1": 0.00077,
            "f1_weighted": 0.000858,
            "precision": 0.001076,
            "precision_weighted": 0.001555,
            "recall": 0.001029,
            "recall_weighted": 0.000969,
            "ap": null,
            "ap_weighted": null
          },
          {
            "accuracy": 0.000848,
            "f1": 0.000567,
            "f1_weighted": 0.001033,
            "precision": 0.001157,
            "precision_weighted": 0.002313,
            "recall": 0.000512,
            "recall_weighted": 0.000848,
            "ap": null,
            "ap_weighted": null
          }
        ],
        "accuracy": 0.000888,
        "f1": 0.000612,
        "f1_weighted": 0.000825,
        "precision": 0.000878,
        "precision_weighted": 0.001499,
        "recall": 0.000917,
        "recall_weighted": 0.000888,
        "ap": NaN,
        "ap_weighted": NaN,
        "main_score": 0.000888,
        "hf_subset": "default",
        "languages": [
          "zxx-Zxxx"
        ]
      }
    ]
  },
  "evaluation_time": 1415.2721869945526,
  "kg_co2_emissions": null,
  "date": 1777633656.245199
}