Dev

[luiztauffer]

• Fix How to properly add velocity or additional features to input pose data in VAME? #189 (extra features #213)
• Fix Update pose loader to use movement's unified load_dataset interface #210 (extra features #213)
• Fix Support passing already-loaded movement datasets to init_new_project #211 (extra features #213)
• Fix Windows compatibility in project init and sample data download #209
• feat: update pose loader to movement 0.16 unified loader interface #212
• docs
• tests passing
• version update
• changelog

@roaldarbol

[codecov]

Codecov Report

❌ Patch coverage is 70.38835% with 61 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/vame/io/load_poses.py	69.84%	19 Missing ⚠️
src/vame/preprocessing/extra.py	73.07%	14 Missing ⚠️
src/vame/preprocessing/to_model.py	28.57%	10 Missing ⚠️
src/vame/initialize_project/new.py	25.00%	6 Missing ⚠️
src/vame/video/video.py	0.00%	5 Missing ⚠️
src/vame/analysis/pose_segmentation.py	42.85%	4 Missing ⚠️
src/vame/io/extra.py	93.61%	3 Missing ⚠️

Files with missing lines	Coverage Δ
src/vame/__init__.py	100.00% <100.00%> (ø)
src/vame/io/__init__.py	100.00% <100.00%> (ø)
src/vame/model/create_training.py	85.38% <100.00%> (+0.34%)	⬆️
src/vame/pipeline.py	86.40% <ø> (ø)
src/vame/preprocessing/__init__.py	100.00% <100.00%> (ø)
src/vame/schemas/project.py	100.00% <100.00%> (ø)
src/vame/util/sample_data.py	93.75% <100.00%> (ø)
src/vame/io/extra.py	93.61% <93.61%> (ø)
src/vame/analysis/pose_segmentation.py	62.54% <42.85%> (-0.52%)	⬇️
src/vame/video/video.py	40.54% <0.00%> (-3.58%)	⬇️
... and 4 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[luiztauffer]

@roaldarbol all changes from the last contributions are now coalesced here. Once you're done with your local tests, we can merge and make a new release

[roaldarbol]

Alright, we're almost there. One thing that seems to be blocking is that I cannot convert the data to egocentric with a single data point, and format_xarray_for_rnn expects centered_reference_keypoint and
orientation_reference_keypoint from ds.attrs. So the only current workaround is to write those manually which feels like a hack, and I think we need a more graceful way of handling that. Here's a minimal example - nc file in the zip for testing.

short_aniframe.zip

import vame

config_path, config = vame.init_new_project(
    project_name="vame_aniframe_demo",
    poses_estimations=["short_aniframe.nc"],
    source_software="movement",
)

Preprocessing is where one needs to specify something extra. This fails:

vame.preprocessing(
    config,
    centered_reference_keypoint="centroid",
    orientation_reference_keypoint="centroid",
)

So a few more need to be set to False:

vame.preprocessing(
    config,
    centered_reference_keypoint="centroid",
    orientation_reference_keypoint="centroid",
    run_egocentric_alignment=False,
    run_savgol_filtering=False,
)

config["extra_features"] = ["area", "eccentricity", "orientation", "solidity"]
vame.util.auxiliary.write_config(config_path, config)
vame.validate_extra_features(config)

The following doesn't work.

vame.create_trainset(config)

It errors out with:

...
KeyError: 'centered_reference_keypoint'

And we need to add the reference points manually despite not really using them.

import os
from pathlib import Path

import xarray as xr

proc = (
    Path(config["project_path"])
    / "data"
    / "processed"
    / f"{config['session_names'][0]}_processed.nc"
)
with xr.open_dataset(proc) as src:
    ds = src.load()
ds.attrs["centered_reference_keypoint"] = "centroid"
ds.attrs["orientation_reference_keypoint"] = "centroid"
ds.to_netcdf(str(proc) + ".tmp", engine="netcdf4")
os.replace(str(proc) + ".tmp", proc)

We should now be able to continue.

vame.create_trainset(config)

vame.train_model(config)
vame.segment_session(config)

[roaldarbol]

Another thing related to the updated unified reader is that we still perform strict input validation although movement also takes care of that. It means that currently, if movement begins supporting other file formats at a later time, they will not be available from VAME despite being legit movement-compatible files. E.g. in neuroinformatics-unit/movement#963, I'm adding support for a special type of Parquet files, but this doesn't translate into being able to use it in VAME.

The blocking parts are in the init_new_project extension whitelist and the ProjectSchema.pose_estimation_filetype Literal.

I think the best way to fix this depends on movement exposing their registries - see #Movement > expose registries - which VAME can then use for the lists, and that way we'll always be up-to-date with their data loaders.

Maybe worth a separate issue.

[roaldarbol]

Actually, it is possible to set run_egocentric_alignment to True and make it work that way it seems:

vame.preprocessing(
    config,
    centered_reference_keypoint="centroid",
    orientation_reference_keypoint="centroid",
    run_egocentric_alignment=True,
    run_savgol_filtering=False,
)

it just creates only NaN's - I don't know if that will result in any issue down the line. But otherwise it seems to run for me.

EDIT: YAY, managed to train a model, do segmentation and plot it all! This was of course just on this single snippet of a file, so next test will be to see how it scales to massive data sets (maybe #184 will need tackling). @luiztauffer I might reach out for a few pointers in the Discussions if that is okay?

[luiztauffer]

@roaldarbol that sounds great!

Where are the NaNs created? VAME excludes the centralized data array from model training, so that's probably why it still works.

I will wait for your larger tests before we merge this to main, feel free to open any Issues or Discussions.

[roaldarbol]

I think the NaN's get produced when making it egocentric, because no angle can be made between the two chosen points - because they are the same. And I think VAME is alright with having only NaN's, but then savgol filtering fails.

As an aside, does VAME have a way of handling a variable where all values are identical or they are NaN's? Let's say a keypoint happened to be at x=1 for the entire session - or x=0 for the centered keypoint? Would they still get used for training and inference in spite of not adding any meaningful information? (This would of course be more realistic to happen in an extra variable).

[roaldarbol]

Just for clarity, here is what the processed nc file looks like:

=== vame_aniframe_demo\data\processed\short_aniframe_processed.nc ===

<xarray.Dataset> Size: 3MB
Dimensions:                      (time: 10000, space: 2, keypoints: 1,
                                  individuals: 1)
Coordinates:
  * time                         (time) int64 80kB 0 1 2 3 ... 9997 9998 9999
  * space                        (space) <U1 8B 'x' 'y'
  * keypoints                    (keypoints) <U8 32B 'centroid'
  * individuals                  (individuals) <U1 4B '1'
Data variables: (12/31)
    position                     (time, space, keypoints, individuals) float64 160kB ...
    confidence                   (time, keypoints, individuals) float64 80kB ...
    area                         (time) float64 80kB ...
    area_convex                  (time) float64 80kB ...
    area_filled                  (time) float64 80kB ...
    axis_major_length            (time) float64 80kB ...
    ...                           ...
    position_cleaned_lowconf     (time, space, keypoints, individuals) float64 160kB ...
    percentage_low_confidence    (space, keypoints, individuals) float64 16B ...
    position_processed           (time, space, keypoints, individuals) float64 160kB ...
    percentage_iqr_outliers      (space, keypoints, individuals) float64 16B ...
    individual_scale             (individuals) float64 8B ...
    position_egocentric_aligned  (time, space, keypoints, individuals) float64 160kB ...
Attributes:
    source_software:                 octron
    ds_type:                         poses
    time_unit:                       frames
    source_file:                     short_aniframe.parquet
    space_unit:                      px
    reference_frame:                 allocentric
    point_of_reference:              bottom_left
    processed_confidence:            True
    processed_outliers:              True
    processed_alignment:             True
    centered_reference_keypoint:     centroid
    orientation_reference_keypoint:  centroid

=== dims ===
  time: 10000
  space: 2
  keypoints: 1
  individuals: 1

=== coords ===
  time (int64, len=10000): [0, 1, 2] ... [9997, 9998, 9999]
  space (<U1): ['x', 'y']
  keypoints (<U8): ['centroid']
  individuals (<U1): ['1']

=== data variables ===
  position                            dims=('time', 'space', 'keypoints', 'individuals') shape=(10000, 2, 1, 1) dtype=float64  min=503.8 max=1199 mean=889.8 n_nan=0
  confidence                          dims=('time', 'keypoints', 'individuals') shape=(10000, 1, 1) dtype=float64  min=0.5001 max=0.9416 mean=0.8417 n_nan=0
  area                                dims=('time',) shape=(10000,) dtype=float64  min=1503 max=4324 mean=2882 n_nan=0
  area_convex                         dims=('time',) shape=(10000,) dtype=float64  min=1680 max=1.021e+04 mean=5447 n_nan=0
  area_filled                         dims=('time',) shape=(10000,) dtype=float64  min=1503 max=4326 mean=2883 n_nan=0
  axis_major_length                   dims=('time',) shape=(10000,) dtype=float64  min=56.78 max=206.4 mean=102.6 n_nan=0
  axis_minor_length                   dims=('time',) shape=(10000,) dtype=float64  min=37.46 max=89.54 mean=59.05 n_nan=0
  bbox_area                           dims=('time',) shape=(10000,) dtype=float64  min=1.16e+04 max=6.859e+04 mean=3.43e+04 n_nan=0
  bbox_aspect_ratio                   dims=('time',) shape=(10000,) dtype=float64  min=0.3424 max=1.995 mean=0.9622 n_nan=0
  eccentricity                        dims=('time',) shape=(10000,) dtype=float64  min=0.1431 max=6.403 mean=1.012 n_nan=0
  equivalent_diameter_area            dims=('time',) shape=(10000,) dtype=float64  min=44.7 max=102.7 mean=61.91 n_nan=0
  euler_number                        dims=('time',) shape=(10000,) dtype=float64  min=-2 max=7 mean=1.216 n_nan=0
  extent                              dims=('time',) shape=(10000,) dtype=float64  min=0.1909 max=5.61 mean=0.568 n_nan=0
  feret_diameter_max                  dims=('time',) shape=(10000,) dtype=float64  min=68.51 max=206.8 mean=118.8 n_nan=0
  moments_hu-0                        dims=('time',) shape=(10000,) dtype=float64  min=0.1923 max=2.274 mean=0.3751 n_nan=0
  moments_hu-1                        dims=('time',) shape=(10000,) dtype=float64  min=1.14e-05 max=1.504 mean=0.04688 n_nan=0
  moments_hu-2                        dims=('time',) shape=(10000,) dtype=float64  min=3.094e-06 max=0.173 mean=0.01188 n_nan=0
  moments_hu-3                        dims=('time',) shape=(10000,) dtype=float64  min=3.242e-08 max=0.1411 mean=0.002041 n_nan=0
  moments_hu-4                        dims=('time',) shape=(10000,) dtype=float64  min=-0.0003329 max=0.01556 mean=2.599e-05 n_nan=0
  moments_hu-5                        dims=('time',) shape=(10000,) dtype=float64  min=-0.003751 max=0.1018 mean=0.000409 n_nan=0
  moments_hu-6                        dims=('time',) shape=(10000,) dtype=float64  min=-0.001315 max=0.0002798 mean=5.675e-07 n_nan=0
  orientation                         dims=('time',) shape=(10000,) dtype=float64  min=-1.571 max=1.571 mean=-0.05226 n_nan=0
  perimeter                           dims=('time',) shape=(10000,) dtype=float64  min=210.7 max=627 mean=420.5 n_nan=0
  perimeter_crofton                   dims=('time',) shape=(10000,) dtype=float64  min=205.1 max=613.4 mean=403.7 n_nan=0
  solidity                            dims=('time',) shape=(10000,) dtype=float64  min=0.3114 max=6.447 mean=0.8118 n_nan=0
C:\Users\Admin\Projects\VAME-Test\inspect_nc.py:32: RuntimeWarning: All-NaN slice encountered
  mn, mx, mean = np.nanmin(arr), np.nanmax(arr), np.nanmean(arr)
C:\Users\Admin\Projects\VAME-Test\inspect_nc.py:32: RuntimeWarning: Mean of empty slice
  mn, mx, mean = np.nanmin(arr), np.nanmax(arr), np.nanmean(arr)
  position_cleaned_lowconf            dims=('time', 'space', 'keypoints', 'individuals') shape=(10000, 2, 1, 1) dtype=float64  min=nan max=nan mean=nan n_nan=20000
  percentage_low_confidence           dims=('space', 'keypoints', 'individuals') shape=(2, 1, 1) dtype=float64  min=100 max=100 mean=100 n_nan=0
  position_processed                  dims=('time', 'space', 'keypoints', 'individuals') shape=(10000, 2, 1, 1) dtype=float64  min=nan max=nan mean=nan n_nan=20000
  percentage_iqr_outliers             dims=('space', 'keypoints', 'individuals') shape=(2, 1, 1) dtype=float64  min=0 max=0 mean=0 n_nan=0
  individual_scale                    dims=('individuals',) shape=(1,) dtype=float64  min=nan max=nan mean=nan n_nan=1
  position_egocentric_aligned         dims=('time', 'space', 'keypoints', 'individuals') shape=(10000, 2, 1, 1) dtype=float64  min=nan max=nan mean=nan n_nan=20000

=== attrs ===
  source_software: 'octron'
  ds_type: 'poses'
  time_unit: 'frames'
  source_file: 'short_aniframe.parquet'
  space_unit: 'px'
  reference_frame: 'allocentric'
  point_of_reference: 'bottom_left'
  processed_confidence: 'True'
  processed_outliers: 'True'
  processed_alignment: 'True'
  centered_reference_keypoint: 'centroid'
  orientation_reference_keypoint: 'centroid'

[luiztauffer]

@roaldarbol if you only have one keypoint, you could safely skip egocentric alignment

[luiztauffer]

@roaldarbol if you only have one keypoint, you could safely skip egocentric alignment

can you confirm this will work for you? If so, I think we could go ahead and merge this for the new release

[roaldarbol]

Sorry I didn't manage to get around to doing final tests for this. Could busy with other things. I'll see if I can give it a spin next week. :-)