Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 7 additions & 12 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,12 @@ DWI_ML
.. image:: https://img.shields.io/badge/License-MIT-yellow.svg
:target: https://github.com/scil-vital/dwi_ml/blob/master/LICENSE

Welcome to the `Sherbrooke Connectivity Imaging Lab (SCIL)`_ and
`Videos & Images Theory and Analytics Laboratory (VITAL)`_ joint DWI_ML
toolkit !
Welcome to the `SCIL`_ and `VITAL`_ joint DWI_ML toolkit !

Links
=====
Documentation
=============

* `Getting started: installation and download <https://dwi-ml.readthedocs.io/en/latest/getting_started.html>`_
* `Complete documentation with a step-by-step process explanation <https://dwi-ml.readthedocs.io/en/latest/>`_
* `Contribute/Submit a patch <https://github.com/scil-vital/dwi_ml/blob/master/CONTRIBUTING.rst>`_
* For the full documentation, including installation and usage, see here: https://dwi-ml.readthedocs.io/en/latest/.
* `Issue tracking <https://github.com/scil-vital/dwi_ml/issues>`_

About
Expand All @@ -40,11 +36,10 @@ file.
Citation
========

If you use DWI_ML in your dMRI data analysis, please cite the toolkit and
provide a link to it.
If you use DWI_ML in your dMRI data analysis, please cite the toolkit and provide a link to it.


.. Links
.. Involved labs
.. _`Sherbrooke Connectivity Imaging Lab (SCIL)`: http://scil.dinf.usherbrooke.ca
.. _`Videos & Images Theory and Analytics Laboratory (VITAL)`: http://vital.dinf.usherbrooke.ca
.. _`SCIL`: http://scil.dinf.usherbrooke.ca
.. _`VITAL`: http://vital.dinf.usherbrooke.ca
Binary file modified docs/_static/images/Learn2track.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/images/logo_dwi_ml_emma.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 4 additions & 0 deletions docs/_static/my_style.css
Original file line number Diff line number Diff line change
Expand Up @@ -67,4 +67,8 @@ footer {

.rst-content li {
padding: 0.1em 0;
}

dt {
margin-bottom: 0 !important;
}
1 change: 1 addition & 0 deletions docs/for_developers/data_management/index.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
.. _data_management_index:

Understanding our data management
=================================
Expand Down
2 changes: 1 addition & 1 deletion docs/for_developers/hdf5/advanced_hdf5_organization.rst
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.. _ref_creating_hdf5:
.. _creating_hdf5:

The hdf5 structure
==================
Expand Down
2 changes: 2 additions & 0 deletions docs/for_developers/models/index.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _create_your_model:

Create your own model
=====================

Expand Down
2 changes: 2 additions & 0 deletions docs/for_developers/testing/general_testing.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _model_testing:

General testing of a model
--------------------------

Expand Down
1 change: 1 addition & 0 deletions docs/for_developers/training/trainers_details.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
.. _trainers_details:

Trainers: the code explained
============================
Expand Down
7 changes: 7 additions & 0 deletions docs/for_users/models/denoising_models.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
.. _denoising_models:


Denoising models
================

Coming soon: Autoencoder (AE) model!
11 changes: 11 additions & 0 deletions docs/for_users/models/our_models.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
.. _our_models:

Our models
==========

.. toctree::
:maxdepth: 1
:caption: Our models

tractography_models
denoising_models
Original file line number Diff line number Diff line change
@@ -1,34 +1,36 @@
.. _our_models:
.. _tractography_models:

Our models
==========
Tractography models
===================

Our library currently only offers two models, both for the task of tracking in the brain, but it could eventually hold models for other tasks.
For more explanation on how to use models for tracking, see :ref:`user_tracking`.

Denoising models
----------------
TractographyTransformers (tt)
*****************************

Coming soon: Autoencoder (AE) model!
This uses transformers and should be the subject of an upcoming publication.

Tractography models
-------------------

For more explanation on how to use models for tracking, see :ref:`user_tracking`.
.. image:: /_static/images/Transformers.png
:align: center
:width: 600


To use this model, run script `tt_track_from_model.py`. . To learn more, run::

tt_track_from_model --help


Learn2track (l2t)
*****************

This is a refactored version of the code prepared by authors of `Poulin2017 <https://link.springer.com/chapter/10.1007/978-3-319-66182-7_62>`_.

.. image:: /_static/images/Learn2track.png
:align: center
:width: 500

To use this model, run script `l2t_track_from_model`.

TractographyTransformers (tt)
*****************************

This uses transformers and should be the subject of an upcoming publication.
To use this model, run script `l2t_track_from_model`. To learn more, run::

ToDo: Add picture
l2t_track_from_model --help

To use this model, run script `tt_track_from_model.py`.
6 changes: 3 additions & 3 deletions docs/getting_started.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Getting started: download and installation
Downloading dwi_ml
******************

To use the DWI_ML toolkit you will need to clone the repository and install the required dependencies::
To use the DWI_ML toolkit you will need to clone the `GitHub repository <https://github.com/scil-vital/dwi_ml>`_ and install the required dependencies::

git clone https://github.com/scil-vital/dwi_ml.git

Expand All @@ -29,12 +29,12 @@ We strongly recommend working in a virtual environment to install all dependenci

**Creating a Comet account**:

- The toolkit uses `comet_ml <https://www.comet.ml/docs/python-sdk/advanced/>`_. It is a python library that creates an "Experiment" (ex, training a model with a given set of hyperparameters) which automatically creates many types of logs online. It requires user to set an API key in $HOME/.comet.config with contents:
- The toolkit uses `comet_ml <https://www.comet.ml/docs/python-sdk/advanced/>`_. It is a python library that creates an "Experiment" (ex, training a model with a given set of hyperparameters) which automatically creates many types of logs online. It requires user to set an API key in $HOME/.comet.config with contents::

| [comet]
| api_key=YOUR-API-KEY

Alternatively, you can add it as an environment variable. Add this to your $HOME/.bashrc file.
Alternatively, you can add it as an environment variable. Add this to your $HOME/.bashrc file::

| export COMET_API_KEY=YOUR-API-KEY

Expand Down
118 changes: 85 additions & 33 deletions docs/index.rst
Original file line number Diff line number Diff line change
@@ -1,43 +1,78 @@
Welcome to DWI_ML documentation!
================================

This website is a guide to the github repository from the SCIL-VITAL organisation: https://github.com/scil-vital/dwi_ml/.
This website is a guide to the github repository from the SCIL-VITAL organisation: https://github.com/scil-vital/dwi_ml/. DWI_ML is a toolkit for Diffusion Magnetic Resonance Imaging (dMRI) analysis
using machine learning and deep learning methods. It is mostly focused on the tractography derivatives of dMRI.


.. image:: /_static/images/logo_dwi_ml_emma_avec_texte.png
:align: center
:width: 500

In this doc, we will present you everything included in this library for you to become either a developer or a user. Note that to get a full understanding of every line of code, you can browse further in each section.

Getting started
---------------
On this page:

- :ref:`section_install`
- :ref:`section_users`
- :ref:`section_advanced_users`
- :ref:`section_developers`

.. _section_install:

1. Installing dwi_ml
--------------------

.. toctree::
:maxdepth: 1
:titlesonly:
:caption: Getting started
:caption: Installing dwi_ml

getting_started

Explanations for users of pre-trained models
--------------------------------------------
.. _section_users:

2. Explanations for users of pre-trained models (Learn2track, Transformers)
---------------------------------------------------------------------------

Pages in this section explain how to use our scripts to use our pre-trained models.

- **Models**: If you want to use our pre-trained models, you may contact us for access to the models learned weights. They will be available online once publications are accepted.
- **1. Downloading models**: If you want to use our pre-trained models, you may contact us for access to the models learned weights. They will be available online once publications are accepted.

- **Using hdf5**: In most cases, data must be organized correctly as a hdf5 before usage. See our page :ref:`hdf5_usage` for an explanation.
- :ref:`our_models`
- :ref:`tractography_models`
- :ref:`denoising_models`

.. toctree::
:maxdepth: 2
:caption: Explanations for users (pre-trained)
- **2. Organizing your data**: In most cases, data must be organized correctly as a hdf5 before usage. Follow the link below for an explanation.

- :ref:`hdf5_usage`

- **3. Using our models to perform tractography**: Use our models to track on your own subjects!

- :ref:`user_tracking`

.. --------------------Hidden toctree: ---------------

for_users/our_models
for_users/hdf5
for_users/tracking

.. toctree::
:maxdepth: 2
:hidden:
:caption: Explanations for users (pre-trained)

Explanations for users of pre-coded models
------------------------------------------
for_users/models/our_models
for_users/hdf5
for_users/tracking

------------------------------

.. _section_advanced_users:

3. Explanations for advanced users: train a model with your own hyperparameters
-------------------------------------------------------------------------------

Pages in this section are useful if you want to train a model based on pre-existing code, such as Learn2track or TractographyTransformers, using your favorite set of hyperparameters.

(Improved documentation coming soon!)

.. toctree::
:maxdepth: 2
Expand All @@ -46,28 +81,45 @@ Pages in this section are useful if you want to train a model based on pre-exist
for_users/from_start_to_finish
for_users/visu_logs

Explanations for developers
---------------------------
.. _section_developers:

4. Explanations for developers: create your own model
-----------------------------------------------------

Page in this section explain more in details how the code is implemented in python.

- **Models**: The first aspect to explore are our models. Discover how you can create your model to fit with our structure. Many parent classes are available for you: if your model inherits from them, they will have access to everything each one offers. For instance, some models have instructions on how to receive inputs from MRI data, prepare inputs in a neighborhood, and use embedding. Other models have access to many options of loss functions for the context of tractography (cosine similarity, classification, Gaussian loss, Fisher von Mises, etc.).
- **1. Create your models**: The first aspect to explore are our models. Discover how you can create your model to fit with our structure. Many parent classes are available for you: if your model inherits from them, they will have access to everything each one offers. For instance, some models have instructions on how to receive inputs from MRI data, prepare inputs in a neighborhood, and use embedding. Other models have access to many options of loss functions for the context of tractography (cosine similarity, classification, Gaussian loss, Fisher von Mises, etc.).

- **Using hdf5**: Our library has been organized to use data in the hdf5 format. Our hdf5 data organization should probably be enough for your needs (see explanations on :ref:`hdf5_usage`), but for more
- :ref:`create_your_model`

- **Training a model**: Then, take a look at how we have implemented our trainers for an efficient management of heavy data.
- **2. Explore our hdf5 organization**: Our library has been organized to use data in the hdf5 format. Our hdf5 data organization should probably be enough for your needs.

- **Using your trained models**: Discover our objects allowing to perform a full tractography from a tractography-model.
- :ref:`hdf5_usage`
- :ref:`creating_hdf5`

- **3. Train your model**: Take a look at how we have implemented our trainers for an efficient management of heavy data. Note that our trainer uses Data Management classes such as our BathLoader and BatchSampler. See below for more information.

.. toctree::
:maxdepth: 1
:caption: Explanations for developers

for_developers/models/index
for_developers/hdf5/advanced_hdf5_organization
for_developers/training/training
for_developers/training/trainers_details
for_developers/data_management/index
for_developers/testing/general_testing
for_developers/testing/tracking_objects
- :ref:`trainers`
- :ref:`trainers_details`
- :ref:`data_management_index`

- **4. Use your trained model**: Discover our objects allowing to perform a full tractography from a tractography-model.

- :ref:`model_testing`
- :ref:`user_tracking`
- :ref:`tracking`

.. --------------------Hidden toctree: ---------------

.. toctree::
:maxdepth: 1
:caption: Explanations for developers
:hidden:

for_developers/models/index
for_developers/hdf5/advanced_hdf5_organization
for_developers/training/training
for_developers/training/trainers_details
for_developers/data_management/index
for_developers/testing/general_testing
for_developers/testing/tracking_objects
File renamed without changes.
26 changes: 13 additions & 13 deletions src/dwi_ml/cli/ae_train_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,19 +17,19 @@
from scilpy.io.utils import (assert_inputs_exist, assert_outputs_exist,
add_verbose_arg)

from dwi_ml.data.dataset.utils import prepare_multisubjectdataset
from dwi_ml.experiment_utils.prints import format_dict_to_str
from dwi_ml.experiment_utils.timer import Timer
from dwi_ml.io_utils import add_memory_args
from dwi_ml.models.projects.ae_models import ModelAE
from dwi_ml.training.trainers import DWIMLTrainer
from dwi_ml.training.utils.batch_samplers import (add_args_batch_sampler,
prepare_batch_sampler)
from dwi_ml.training.utils.batch_loaders import (add_args_batch_loader)
from dwi_ml.training.utils.trainer import (add_training_args, run_experiment,
format_lr)
from dwi_ml.training.batch_loaders import DWIMLStreamlinesBatchLoader
from dwi_ml.training.utils.experiment import (
from dwi_ml.general.data.dataset.utils import prepare_multisubjectdataset
from dwi_ml.general.experiment_utils.prints import format_dict_to_str
from dwi_ml.general.experiment_utils.timer import Timer
from dwi_ml.general.io_utils import add_memory_args
from dwi_ml.projects.AE.ae_models import ModelAE
from dwi_ml.general.training.trainers import DWIMLTrainer
from dwi_ml.general.training.utils.batch_samplers import (add_args_batch_sampler,
prepare_batch_sampler)
from dwi_ml.general.training.utils.batch_loaders import (add_args_batch_loader)
from dwi_ml.general.training.utils.trainer import (add_training_args, run_experiment,
format_lr)
from dwi_ml.general.training.batch_loaders import DWIMLStreamlinesBatchLoader
from dwi_ml.general.training.utils.experiment import (
add_mandatory_args_experiment_and_hdf5_path)


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@
from scilpy.io.utils import assert_inputs_exist, assert_outputs_exist, \
add_verbose_arg, add_overwrite_arg

from dwi_ml.data.hdf5.utils import format_nb_blocs_connectivity
from dwi_ml.data.processing.streamlines.post_processing import \
from dwi_ml.general.data.hdf5.utils import format_nb_blocs_connectivity
from dwi_ml.general.data.processing.streamlines.post_processing import \
compute_triu_connectivity_from_blocs, \
find_streamlines_with_chosen_connectivity, prepare_figure_connectivity

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
from scilpy.io.utils import assert_inputs_exist, assert_outputs_exist, \
add_verbose_arg, add_overwrite_arg

from dwi_ml.data.processing.streamlines.post_processing import \
from dwi_ml.general.data.processing.streamlines.post_processing import \
find_streamlines_with_chosen_connectivity, \
compute_triu_connectivity_from_labels, prepare_figure_connectivity

Expand Down
6 changes: 3 additions & 3 deletions src/dwi_ml/cli/dwiml_create_hdf5_dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,10 @@

from dipy.io.stateful_tractogram import set_sft_logger_level

from dwi_ml.data.hdf5.hdf5_creation import HDF5Creator
from dwi_ml.data.hdf5.utils import (
from dwi_ml.general.data.hdf5.hdf5_creation import HDF5Creator
from dwi_ml.general.data.hdf5.utils import (
add_hdf5_creation_args, add_streamline_processing_args)
from dwi_ml.experiment_utils.timer import Timer
from dwi_ml.general.experiment_utils.timer import Timer


def _initialize_intermediate_subdir(hdf5_file, save_intermediate):
Expand Down
Loading
Loading