Skip to content

dataset_preparation.md

Latest commit

 

History

History
113 lines (75 loc) · 4.76 KB

dataset_preparation.md

File metadata and controls

113 lines (75 loc) · 4.76 KB
Raw

Dataset Preparation Guide

This guide provides detailed instructions for obtaining and preparing all datasets used in the CLEF project. All datasets are available through PhysioNet, except for AuroraBP, which can be downloaded upon request.

1. Dataset Overview

Dataset Size URL
PTB-XL ~3.2 GB physionet.org/content/ptb-xl/1.0.3/
MIMIC-IV-ECG ~90.4 GB physionet.org/content/mimic-iv-ecg/1.0/
Chapman ~5.4 GB physionet.org/content/ecg-arrhythmia/1.0.0/
Icentia11k ~1.1 TB physionet.org/content/icentia11k-continuous-ecg/1.0/
Aurora-BP ~21 GB github.com/microsoft/aurorabp-sample-data
MC-MED ~757 GB physionet.org/content/mc-med/1.0.1/
MUSIC ~90.2 GB physionet.org/content/music-sudden-cardiac-death/1.0.1/

2. Data Preprocessing Instructions

2.1 Download All Datasets

Please download all required datasets to your local directory for model evaluation.

For model pretraining, you will also need to download the noise data file DATA_noises_real.mat from this PhysioNet link and place it in the dataset folder.

2.2 Datasets That Need Processing

Two of the datasets, Chapman and Icentia11k require additional processing.

For MIMIC-IV, AuroraBP, MUSIC, MC-MED and PTB-XL, no processing is needed.

2.2.1 Chapman Dataset Processing

To prepare the Chapman dataset, please follow these steps:

a. CSV Files
The following files that mark the train/val/test split are included in this repository (originally from the HeartLang repository) and are located in dataset/chapman/:

  • chapman_train.csv
  • chapman_val.csv
  • chapman_test.csv

b. Preprocessing
Run the preprocessing script (a modified version from HeartLang that keeps the original sampling rate of 500 Hz) to generate corresponding data files:

python preprocess/preprocess_chapman.py

After running the preprocessing script, a data folder will be created inside the dataset/chapman directory. This folder will contain the processed .npy files for the train, validation, and test splits. Your directory structure should look like the following:

dataset/
└── chapman/
    ├── chapman_train.csv
    ├── chapman_val.csv
    ├── chapman_test.csv
    └── data/
        ├── train_data.npy
        ├── train_labels.npy
        ├── val_data.npy
        ├── val_labels.npy
        ├── test_data.npy
        ├── test_labels.npy

2.2.2 Icentia11k Dataset Processing

To preprocess the Icentia11k dataset for both beat and rhythm tasks, use the provided script with the following instructions:

Run the Preprocessing Script

Execute the script with the desired arguments. For example, to preprocess both beat and rhythm tasks:

python preprocess/preprocess_icentia.py --data_path <path_to_icentia_data> --output_dir dataset/Icentia11K --task all
  • --data_path: Path to the Icentia data directory
  • --output_dir: Output directory for preprocessed data
  • --task: Which task to preprocess data for (beat, rhythm, or all)

3. Datasets That Do Not Require Processing

For the following datasets, no data processing is required. You only need to specify the correct data file in config/dataset.

  • MC-MED:
    The processed labels for MC-MED are provided in dataset/mcmed/variables_encounter_all.csv.

  • MIMIC-IV (labels from ECGFounder):
    For MIMIC-IV, we use the labels generated by ECGFounder, which are available in dataset/mimic-iv/LVEF.csv.

  • MUSIC:
    The outcome labels for the MUSIC dataset are stored in dataset/music/standard-outcome-variables.csv.
    Some records are missing from the dataset; a list of these missing files can be found in dataset_ori/music/music_missing_files.txt.

  • PTB-XL:
    No additional processing is needed for PTB-XL. The files raw100.npy and raw500.npy will be generated automatically under dataset/ptbxl.

  • AuroraBP:
    No extra processing is required for AuroraBP.