Audiovisual dataset loader

.github/environment-ci.yml

@@ -39,4 +39,5 @@ dependencies:
     - smart_open[all]>=6.4.0
     - testcontainers>=3.7.1
     - mir_eval>=0.7
-    - moviepy>=2.2.1
+    - moviepy>=2.2.1
+    - opencv-python

docs/index.rst

@@ -76,6 +76,3 @@ Please see :ref:`contributing` for guidelines.
    :maxdepth: 1
 
    source/faq
-
-
-

docs/source/mirdata.rst

@@ -433,6 +433,14 @@ salami
    :inherited-members:
 
 
+saraga_audiovisual
+^^^^^^^^^^^^^^^^^^
+
+.. automodule:: mirdata.datasets.saraga_audiovisual
+    :members:
+    :inherited-members:
+
+
 saraga_carnatic
 ^^^^^^^^^^^^^^^
 
@@ -543,6 +551,8 @@ Annotation Types
    :members:
 .. autoclass:: mirdata.annotations.F0Data
    :members:
+.. autoclass:: mirdata.annotations.GestureData
+    :members:
 .. autoclass:: mirdata.annotations.MultiF0Data
    :members:
 .. autoclass:: mirdata.annotations.NoteData
@@ -591,4 +601,4 @@ mirdata.download_utils
 ^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: mirdata.download_utils
-   :members:
+   :members:

docs/source/quick_reference.rst

@@ -10,7 +10,7 @@
 Supported Datasets and Annotations
 ##################################
 
-Dataset Quick Reference 
+Dataset Quick Reference
 =======================
 
 This table is provided as a guide for users to select appropriate datasets. The
@@ -51,8 +51,8 @@ types are implemented in Mirdata see :ref:`annotations`.
 
 Events
 ^^^^^^
-A generic annotation to indicate whether a particular event is happening at a given time. 
-It can be used, for instance, to indicate whether a particular instrument is playing at a 
+A generic annotation to indicate whether a particular event is happening at a given time.
+It can be used, for instance, to indicate whether a particular instrument is playing at a
 given time-step or whether a particular note is being played at a given time-step. In fact,
 it is implicit in annotations such as F0_ or Vocal Notes_ (instrument is activated when the
 melody is non-0). However, some datasets provide it as a standalone event annotation.
@@ -92,7 +92,7 @@ which may be encoded in a number of formats/granularities, including midi note n
 
 Effect
 ^^^^^^
-Effect applied to a track. It may refer to the effect applied to a single stroke or an entire track. 
+Effect applied to a track. It may refer to the effect applied to a single stroke or an entire track.
 It can include the effect name, the effect type, the effect parameters, and the effect settings.
 
 .. _genre:
@@ -102,6 +102,12 @@ Genre
 A typically global "tag", indicating the genre of a recording. Note that the concept of genre is highly
 subjective and we refer those new to this task to this `article`_.
 
+.. _gesture:
+
+Gesture
+^^^^^^^
+Keypoints and confindence scores of the pose estimation of a musician while playing an instrument.
+
 .. _instruments:
 
 Instruments

docs/source/table.rst

@@ -124,6 +124,7 @@
           :target: https://creativecommons.org/licenses/by-nc-sa/4.0
 
    * - .. line-block::
+
         (CompMusic)
         Indian Tonic
 
@@ -253,9 +254,9 @@
      - .. image:: https://licensebuttons.net/l/by-sa/4.0/80x15.png
           :target: https://creativecommons.org/licenses/by-sa/4.0
 
-   * - .. line-block::
+   * - .. ::
 
-       Freesound One-Shot 
+       Freesound One-Shot
        Percussive Sounds
 
      - - audio: ✅
@@ -392,6 +393,18 @@
      - .. image:: https://img.shields.io/badge/License-MIT-blue.svg
           :target: https://lbesson.mit-license.org/
 
+   * - .. ::
+
+       MTG Jamendo
+       Autotagging Moodtheme
+
+     - - audio: ✅
+       - annotations: ✅
+     - - moodtheme annotations
+     - 18448
+     - .. image:: https://licensebuttons.net/l/by-nc-sa/4.0/80x15.png
+          :target: https://creativecommons.org/licenses/by-nc-sa/4.0
+
    * - MAESTRO
      - - audio: ✅
        - annotations: ✅
@@ -442,9 +455,9 @@
      - .. image:: https://licensebuttons.net/l/by/3.0/80x15.png
           :target: https://creativecommons.org/licenses/by/3.0/
 
-   * - .. line-block::
+   * - .. ::
 
-       MTG Jamendo 
+       MTG Jamendo
        Autotagging Moodtheme
 
      - - audio: ✅
@@ -530,6 +543,14 @@
      - .. image:: https://licensebuttons.net/l/zero/1.0/80x15.png
           :target: http://creativecommons.org/publicdomain/zero/1.0/
 
+   * - Saraga Audiovisual
+     - - audio: ✅
+         - annotations: ✅
+     - - :ref:`gesture`
+     - 233
+     - .. image:: https://licensebuttons.net/l/by-nc-sa/4.0/80x15.png
+          :target: https://creativecommons.org/licenses/by-nc-sa/4.0
+
    * - Saraga Carnatic
      - - audio: ✅
        - annotations: ✅
@@ -556,9 +577,9 @@
      - 108
      - .. image:: https://licensebuttons.net/l/by-nc-sa/4.0/80x15.png
           :target: https://creativecommons.org/licenses/by-nc-sa/4.0
-          
-   * - .. line-block::
-   
+
+   * - .. ::
+
        Saraga Carnatic
        Melody Synth (SCMS)
 

mirdata/annotations.py

@@ -204,6 +204,22 @@ def __init__(
         self.confidence = confidence
 
 
+class GestureData(Annotation):
+    """GestureData Class
+
+    Attributes:
+        keypoints (np.ndarray):
+        scores (np.ndarray):
+    """
+
+    def __init__(self, keypoints, scores):
+        validate_array_like(keypoints, np.ndarray, np.float32)
+        validate_array_like(scores, np.ndarray, np.float32)
+
+        self.keypoints = keypoints
+        self.scores = scores
+
+
 class F0Data(Annotation):
     """F0Data class