Skip to content

Add MLX multimodal inference support#10383

Open
jbg wants to merge 1 commit into
mainfrom
jbg/mlx-multimodal-safemlx-update
Open

Add MLX multimodal inference support#10383
jbg wants to merge 1 commit into
mainfrom
jbg/mlx-multimodal-safemlx-update

Conversation

@jbg

@jbg jbg commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • update the safemlx dependency family to the latest compatible releases
  • enable image processing for MLX models and route ordered image/text inputs through model processors
  • preserve text-only generation and disable draft-model generation when media is present
  • require processor metadata when advertising supported multimodal MLX repositories
  • add focused coverage for prompt ordering, processor requirements, marker validation, and text-only behavior

Why

Recent safemlx releases add multimodal model processors and change prepared-input generation APIs. Goose needs to use those APIs so MLX vision models receive decoded image data alongside the rendered prompt while text-only models continue using the existing token input path.

Validation

  • cargo fmt
  • cargo check -p goose-local-inference --features mlx
  • cargo test -p goose-local-inference --features mlx (130 tests passed)
  • cargo check -p goose-local-inference

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 29a59a24a8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +149 to +153
Some(
loaded
.model
.prepare_input(&processor_input)
.map_err(mlx_error)?,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Account for media tokens before capping generation

When has_media is true, this branch prepares a processor input that can expand each image into additional model positions, but the earlier context check and the later mlx_max_tokens(..., prompt_tokens.len()) still use the plain text token count from the marker-only prompt. For image requests near the model context limit, Goose can allow a prompt that only fits before image expansion and then either fail during generation or overrun the intended context budget; use the prepared input's effective length or an image-token estimate when validating and computing headroom.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant