Emit progress_total events from PreTrainedModel.from_pretrained()

[JacobiusMakes]

Summary

Closes #1052.

When users call PreTrainedModel.from_pretrained() directly (rather than through pipeline()), they currently have no way to know the total size of all model files before downloads begin. This makes it impossible to render a single aggregate progress bar.

This PR adds the same progress_total event infrastructure that already exists in pipeline() to PreTrainedModel.from_pretrained():

• After resolving the model config but before starting downloads, gathers file metadata (sizes) for all expected model files via lightweight Range/HEAD requests using the existing get_model_files() and get_file_metadata() utilities
• Wraps the user's progress_callback to emit progress_total events on each progress update, containing aggregate loaded/total byte counts plus a per-file breakdown in files
• Follows the exact same pattern already used in pipelines.js (lines 139-178)
• Fails gracefully: if metadata fetching fails (e.g., local-only models), downloads proceed normally without total progress tracking

Example usage

const model = await AutoModel.from_pretrained('model-id', {
  progress_callback: (info) => {
    if (info.status === 'progress_total') {
      // info.progress  — 0..100 across all files
      // info.loaded    — total bytes loaded so far
      // info.total     — total bytes expected
      // info.files     — per-file { loaded, total } map
      updateProgressBar(info.progress);
    }
  },
});

Test plan

• Verified prettier formatting passes
• Ran tests/utils/hub.test.js — all pass
• Ran tests/utils/model_registry.test.js — all pass
• Syntax check (node -c) passes
• Manual test with a multi-file model to verify progress_total events fire correctly

Changes

packages/transformers/src/models/modeling_utils.js — Added ~60 lines to PreTrainedModel.from_pretrained() that gather file metadata upfront and wrap the progress callback to emit progress_total events.

🤖 Generated with Claude Code

[nico-martin]

I ran into the exact same ptoblem too yesterday :D
Thank you so much for making a PR. Will check it out asap!

[nico-martin]

Thanks for this PR, I like the direction. I noticed two issues while testing that I think we should fix before merging:

1. Total progress will always stay below 100%

• config.json is loaded first.
• Then the total-progress tracker is built and includes config.json in the total bytes.
• But that file is already finished, so its loaded bytes are never added to the new tracker.

So in practice, the final progress_total can get stuck under 100% even when all downloads are done.

2. Duplicate progress_total events when using pipeline()

• pipeline() already emits aggregated progress_total.
• This PR also emits progress_total from model loading.
• In pipeline(), both streams are forwarded/emitted, so clients receive both.

So users can see about 2x progress_total events for model download.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[JacobiusMakes]

Thanks for the thorough review! Both issues fixed in e28ce2c:

1) config.json progress stuck below 100% — config.json is now pre-marked as fully loaded in files_loading since AutoConfig.from_pretrained() fetches it before the tracker is set up.

2) Duplicate progress_total with pipeline() — Wrapped callbacks now carry a _progress_total_wrapped flag. from_pretrained() checks for it and skips wrapping if pipeline() already set it up. pipeline()'s wrapper also sets this flag for consistency.

[nico-martin]

There is still a problem with the prod build:

tsc --build

Error: src/models/modeling_utils.js(430,53): error TS2339: Property '_progress_total_wrapped' does not exist on type 'ProgressCallback'.
Error: src/pipelines.js(180,33): error TS2339: Property '_progress_total_wrapped' does not exist on type '(info: ProgressInfo) => void'.
 ELIFECYCLE  Command failed with exit code 1.
/home/runner/work/transformers.js/transformers.js/packages/transformers:
 ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL  @huggingface/transformers@4.0.0-next.10 build: `node scripts/build.mjs && pnpm typegen`
Exit status 1
 ELIFECYCLE  Command failed with exit code 1.
Error: Process completed with exit code 1.

https://github.com/huggingface/transformers.js/actions/runs/23668078336/job/69010757045?pr=1609

[xenova]

thanks for the PR! I think a better way to handle this is with a wrapper function class instance which can detect if we need to wrap the object using an instanceof test. this means we don't need to attach hacky properties like _progress_total_wrapped to it.

[xenova]

merged via #1615