Skip to content

use more robust validation, defined at dataset level #806

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ShriyaRishab merged 4 commits into from
Jul 25, 2025
Merged

use more robust validation, defined at dataset level #806

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
20e83ee
use more robust validation, defined at dataset level
CarlosGomes98 Jul 23, 2025
2789097
Merge branch 'master' into flux/robust_validation
CarlosGomes98 Jul 23, 2025
91cdaa7
clearer readme
CarlosGomes98 Jul 23, 2025
3e11b7c
change name to flux1, add randomness info to readme
CarlosGomes98 Jul 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions text_to_image/README.md
View file
Open in desktop

@ShriyaRishab ShriyaRishab Jul 23, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid confusion, can we update the pseudocode in Quality metric section so t is not computed but is obtained directly from the dataset? It will help submitters to see pseudocode for what they need to implement instead of how the validation dataset was originally generated.

You can add an appendix in the end with the pseudocode used to generate the timestamps for the validation dataset as an FYI

Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,8 @@ The filter file is included in this repository. It was generated using `torchtit

### COCO-2014 subset

For validation purposes, each sample of the dataset is associated with a timestep that is used to evaluate it.
For more details, consult the [evaluation algorithm](#quality-metric)
To download the cleaned data, run the following:

**Note:** We reccomend training directly on preprocessed embeddings. To do that, skip [here](#preprocessing).
Expand Down Expand Up @@ -288,6 +290,9 @@ For precise details, we encourage the consultation of the code at `torchtitan/ex
# 5. Quality
### Quality metric
Validation loss averaged over 8 equidistant time steps [0, 7/8], as described in [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://arxiv.org/pdf/2403.03206).
The validation dataset is prepared in advance so that each sample is associated with a timestep.
This is an integer from 0 to 7 inclusive, and thus should be divided by `8.0` to obtain the timestep.

The algorithm is as follows:

```pseudocode
Expand All @@ -314,6 +319,9 @@ validation_loss = mean(mean_per_timestep)
RETURN validation_loss
```

As we ensure that the validation set has an equal number of samples per timestep,
a simple average of all loss values is equivalent to the above.

### Quality target
0.586
### Evaluation frequency
Expand Down
2 changes: 1 addition & 1 deletion text_to_image/torchtitan
Open in desktop
Submodule torchtitan updated 10 files
+66 −17 README.md
+10 −11 torchtitan/components/metrics.py
+12 −10 torchtitan/experiments/flux/dataset/flux_mlperf.py
+2 −1 torchtitan/experiments/flux/mlperf_logging.py
+4 −4 torchtitan/experiments/flux/parallelize_flux.py
+9 −2 torchtitan/experiments/flux/scripts/coco_to_webdataset.py
+23 −15 torchtitan/experiments/flux/scripts/preprocess_flux_dataset.py
+34 −90 torchtitan/experiments/flux/train.py
+1 −1 torchtitan/experiments/flux/train_configs/debug_model.toml
+4 −1 torchtitan/experiments/flux/utils.py