Skip to content

Sync main to release (v1.1.2)#302

Open
Erikpostt wants to merge 121 commits into
releasefrom
sync-main-to-release
Open

Sync main to release (v1.1.2)#302
Erikpostt wants to merge 121 commits into
releasefrom
sync-main-to-release

Conversation

@Erikpostt

@Erikpostt Erikpostt commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Release v1.1.2

Release date: 15/06/2026

Description

This PR synchronizes the main branch into release to prepare for the v1.1.2 release of ParaDigMa.

Changes

Breaking Changes

  • Unified pipeline return format: All pipelines (run_gait_pipeline(), run_tremor_pipeline(), run_pulse_rate_pipeline()) now return a unified dictionary structure instead of tuples/DataFrames:
    • Previously:
      • Gait: (quantified_df_dict: dict, metadata_dict: dict) (from v1.1.1)
      • Tremor/Pulse Rate: DataFrame
    • Now: All return unified dict with keys:
      • 'quantification': Computed results (dict for gait, DataFrame for others)
      • 'metadata': Metadata dict (gait only)
      • 'preprocessing', 'classification': Intermediate results (if requested via return_intermediate)
      • '_steps_executed': List of steps that were actually executed
      • '_error': Error message (None if successful)

Features

  • Selective pipeline step execution: New parameters run_steps and return_intermediate in all pipeline functions and orchestrator:

    • run_steps: Which processing steps to execute ('preprocessing', 'classification', 'quantification', 'aggregation')
    • return_intermediate: Which intermediate results to include in return dict (independent of disk storage)
    • Available in both individual pipeline functions and main run_paradigma() orchestrator
    • Enables parallel processing workflows where preprocessing/classification run separately from quantification
    • Example: run_steps=['preprocessing', 'classification'] to stop after classification without computing quantification
  • Adaptive sampling frequency detection: orchestrator now automatically detects data sampling frequency and adjusts frequency-dependent parameters accordingly. It is no longer necessary to manually set the sampling frequency to 100 Hz

  • Independent save/return control: save_intermediate and return_intermediate now work independently:

    • Save results to disk without returning them (memory-efficient for large datasets)
    • Return intermediate results without saving to disk (useful for debugging)
    • Previously, saving automatically returned results in memory
  • Orchestrator updates: run_paradigma() now supports selective step execution and handles unified dict returns from all pipelines:

    • Updated to work with new pipeline return formats (unified dict structure)
    • Properly handles gait filtered/unfiltered results in combined output
    • Enables conditional execution paths (e.g., only run quantification for pre-processed data)

Improvements

  • Added comprehensive error handling with _error field in all pipeline function returns.
  • All pipeline functions now log which steps are being executed and returned.

Checklist

  • Tests passed
  • Documentation updated
  • Version bumped in pyproject.toml (1.1.1 → 1.1.2)

KarsVeldkamp and others added 30 commits February 27, 2025 13:14
* Remove capitalization and rename gait to arm swing during gait

* Change probability to predictions and update arrows in tremor pipeline

* Change probability to predictions and update arrows in tremor pipeline

---------

Co-authored-by: nienketimmermans <65012819+nienketimmermans@users.noreply.github.com>
The badge stayed at `3.10+` while the requirement changed to `3.11+`. A small update for consistency
Adding static pages for github pages to render
Erikpostt and others added 24 commits February 4, 2026 09:28
* Update dependencies, bump to v1.1.0

* Update citation and changelog
* Solve conflicts
* Resolve merge conflicts with release
* Add unfiltered gait quantification to store both clean and all gait data

BREAKING CHANGE: run_gait_pipeline() now returns nested dicts instead of flat tuple

Previously, run_gait_pipeline() returned:
  (quantified_df: DataFrame, metadata: dict)

Now returns:
  ({'filtered': df, 'unfiltered': df}, {'filtered': meta, 'unfiltered': meta})

This change allows analysis of both:
- Filtered (clean gait only): Arm swing during walking without other activities
- Unfiltered (all gait): All arm swing during any gait, regardless of quality

Changes:
- gait_pipeline.py: Call quantify_arm_swing() twice (filtered & unfiltered)
- gait_pipeline.py: Fix early returns to match new return structure
- orchestrator.py: Update result structure to handle nested gait dicts
- orchestrator.py: Separate Step 4 concatenation and Step 5 aggregation loops
- orchestrator.py: Aggregate both filtered and unfiltered quantifications
- orchestrator.py: Save both types of quantifications and aggregations
- test_gait_analysis.py: Update test assertions for new return structure

Motivation: Previous version only stored filtered quantification, losing
unfiltered data. This made it impossible to compare total gait duration
vs clean gait duration or analyze arm swing characteristics across all gait.

* Fix orchestrator to pass required segment_meta and segment_cats to aggregate_arm_swing_params

- Added parsing of segment_length_bins to segment_cats tuples
- Pass segment_meta from metadata dictionary to aggregation function
- Applies to both filtered and unfiltered gait aggregations
#267)

* Clean up test infrastructure: remove unused test files, notebooks, and consolidate helpers to conftest.py
#269)

* Fix save_intermediate parameter: use 'classification' for all prediction outputs and store predictions instead of features
* Bump version to 1.1.1 and update changelog
* Sync release into main
* modify spectrogram computation
…277)

* Added option to run specific steps and return specific output. Updated docs accordingly
* test: Add datetime implementation test suite and verification
* fix: Ignore unused start_dt in load_data_files
* docs: Add backward compatibility analysis and comparison tools
* Fix timestamp precision bug: add 1/fs to end_dt calculations

Both combined and per_segment end_dt calculations now account for sample duration.
Previously, end_dt used time_array.max() which represents the START of the last
sample, not the END. Adding 1/fs (0.01s for 100Hz data) gives the correct end time.

Fixes: segment durations now match duration_s values (e.g., segment 1 is 5.25s
with end_dt 0.01s later)

* Fix: Preserve timezone in quantification metadata when start_dt is passed with timezone

* Ensure backward compatibility by including start_s and end_s EVEN in case of start_st
* fix: improve error handling and logging in gait pipeline

- Add logging parameter to extract_arm_activity_features
- Add logging parameter to quantify_arm_swing
- Fix silent failure: raise ValueError on no windows instead of empty return
- Replace print() with logger.warning() for proper log capture
- Add context to errors: segment type, duration, data shape
- Pass logger through function call chain in run_gait_pipeline

When feature extraction fails or no data creates windows, now:
- Error is logged to file with full context
- Error includes segment/filter info and duration
- Pipeline recognizes failure immediately
- Users can trace errors back to specific subjects/files/segments
* fix: preserve per_segment metadata structure in gait pipeline orchestrator
* Fix: Use separate fields for segment categorization vs summation

- duration_for_categorization_s: unfiltered parent segment duration
- duration_s: actual segment data duration (filtered or unfiltered)

Filtered arm swing of 5s in 25s unfiltered segment now correctly:
- Categorizes into 20+s (based on parent unfiltered duration)
- Sums only 5s to category total (actual filtered data)

* Ensured gait categories are always scoped by unfiltered gait
* chore: update dependencies to patch security vulnerabilities

- Update tornado to >=6.4.1 (fixes DoS and cookie injection vulnerabilities)
- Update mistune to >=3.0.2 (fixes ReDoS in LINK_TITLE_RE)
- Update pytest to >=7.4.4 (fixes tmpdir handling vulnerability)
- Update jupyter-server to 2.18.2 (fixes CORS bypass, path traversal)
- Update jupyterlab to 4.5.7 (fixes XSS vulnerabilities)
- Update nbconvert to >=7.17.0 (fixes path traversal in cell attachments)
- Update requests to >=2.32.6 (fixes insecure temp file reuse)
- Update multiple dev dependencies to patch aiohttp issues

Addresses 26 open security vulnerabilities from Dependabot scan

* Reset jupyterlab-widgets to <= 3.0.15 because of installation errors for >= 3.0.16
…tment (#285)

* feat: Add adaptive frequency bounds for variable sampling rates

- Add sampling_frequency property with setter to IMUConfig
- Implement _update_frequency_dependent_params() virtual method
- Override in GaitConfig: auto-clamp spectrum_high_frequency and mfcc_high_frequency to Nyquist
- Override in TremorConfig: auto-clamp fmax_peak_search and fmax_mfcc to Nyquist
- Fixes issue where 50Hz/64Hz data caused 'out of bounds' errors in feature extraction

This allows users to set sampling_frequency at any point and freq-dependent params
adapt automatically, preventing Nyquist violations without manual config management.

* fix: Only clamp MFCC bounds when they exceed Nyquist, preserve 25 Hz band
* prerelease adjustments

* Update version date 

Updated version date for v1.1.2 and modified breaking changes section.
* Release v1.0.3

* Sync main to release (#233)

* Incorporate modal tremor power in aggregate function

* Add tolerance for contiguous data to config

* Add pre-commit hooks and add Poetry scripts for simplifying dev process

* Remove segment categorization (set to legacy)

* Adjust pipelines to flexible column names and config updates

* Add info on scale sensitivity and make accelerometer optional

* Clarify installing pre-commit

* Release v1.1.0 (#255)

* Merge dev branch into release

* Remove openmovement dependency (#256)

* Remove openmovement dependency

* Solve openmovement pypi conflict (#257)

* Solve dependency conflicts and PyPi issue

* Release v1.1.1 (#272)

* Add unfiltered gait quantification to store both clean and all gait data

* Fix orchestrator to pass required segment_meta and segment_cats to aggregate_arm_swing_params

* Clean up test infrastructure: remove unused test files, notebooks, and consolidate helpers to conftest.py

* Fix save_intermediate parameter: use 'classification' for all prediction outputs and store predictions instead of features

* Bump version to 1.1.1 and update changelog

* Remove tests/notebooks/ folder

---------

Co-authored-by: Erik Post <57133568+Erikpostt@users.noreply.github.com>
* Release v1.0.3

* Sync main to release (#233)

* Incorporate modal tremor power in aggregate function

* Add tolerance for contiguous data to config

* Add pre-commit hooks and add Poetry scripts for simplifying dev process

* Remove segment categorization (set to legacy)

* Adjust pipelines to flexible column names and config updates

* Add info on scale sensitivity and make accelerometer optional

* Clarify installing pre-commit

* Release v1.1.0 (#255)

* Merge dev branch into release

* Remove openmovement dependency (#256)

* Remove openmovement dependency

* Solve openmovement pypi conflict (#257)

* Solve dependency conflicts and PyPi issue

* Release v1.1.1 (#272)

* Add unfiltered gait quantification to store both clean and all gait data

* Fix orchestrator to pass required segment_meta and segment_cats to aggregate_arm_swing_params

* Clean up test infrastructure: remove unused test files, notebooks, and consolidate helpers to conftest.py

* Fix save_intermediate parameter: use 'classification' for all prediction outputs and store predictions instead of features

* Bump version to 1.1.1 and update changelog

* Remove tests/notebooks/ folder

* fix unintended changes

---------

Co-authored-by: Erik Post <57133568+Erikpostt@users.noreply.github.com>
@Erikpostt Erikpostt marked this pull request as ready for review June 15, 2026 08:53
@Erikpostt Erikpostt self-assigned this Jun 15, 2026
@Erikpostt

Copy link
Copy Markdown
Contributor Author

@KarsVeldkamp @nienketimmermans Note that this is a workaround for the merge conflicts in #299. To reproduce:

  1. Pull release
  2. Create a new branch from release
  3. Merge main into this new branch (locally)
  4. Push new branch and merge into release

This seems to solve the diverging histories. I suggest after the release to remove the main branch and re-create it from release, such that the histories align. Let's do a little bit of research this does not break stuff.

@Erikpostt Erikpostt changed the title Sync main to release Sync main to release (v1.1.2) Jun 15, 2026
@Erikpostt Erikpostt requested a review from a team June 15, 2026 14:44

@nienketimmermans nienketimmermans left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! @KarsVeldkamp let's discuss whether we want to delete the main branch or if this workaround is good enough for future releases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants