Merge branch 'main' into videollama3

Author: lkhl

docs/source/en/_toctree.yml

@@ -277,6 +277,8 @@
         title: Keypoint detection
       - local: tasks/knowledge_distillation_for_image_classification
         title: Knowledge Distillation for Computer Vision
+      - local: tasks/keypoint_matching
+        title: Keypoint matching
       title: Computer vision
     - sections:
       - local: tasks/image_captioning

docs/source/en/kv_cache.md

@@ -102,7 +102,7 @@ You may want to consider offloading if you have a small GPU and you're getting o
 Offloading is available for both [`DynamicCache`] and [`StaticCache`]. You can enable it by configuring `cache_implementation="offloaded"` for the dynamic version, or `cache_implementation="offloaded_static"` for the static version, in either [`GenerationConfig`] or [`~GenerationMixin.generate`].
 Additionally, you can also instantiate your own [`DynamicCache`] or [`StaticCache`] with the `offloading=True` option, and pass this cache in `generate` or your model's `forward` (for example, `past_key_values=DynamicCache(config=model.config, offloading=True)` for a dynamic cache).
 
-Note that the 2 [`Cache`] classes mentionned above have an additional option when instantiating them directly, `offload_only_non_sliding`.
+Note that the 2 [`Cache`] classes mentioned above have an additional option when instantiating them directly, `offload_only_non_sliding`.
 This additional argument decides if the layers using sliding window/chunk attention (if any), will be offloaded as well. Since
 these layers are usually short anyway, it may be better to avoid offloading them, as offloading may incur a speed penalty. By default, this option is `False` for [`DynamicCache`], and `True` for [`StaticCache`].
 

docs/source/en/model_doc/efficientloftr.md

@@ -148,6 +148,14 @@ processed_outputs = processor.post_process_keypoint_matching(outputs, image_size
 - post_process_keypoint_matching
 - visualize_keypoint_matching
 
+## EfficientLoFTRImageProcessorFast
+
+[[autodoc]] EfficientLoFTRImageProcessorFast
+
+- preprocess
+- post_process_keypoint_matching
+- visualize_keypoint_matching
+
 <frameworkcontent>
 <pt>
 ## EfficientLoFTRModel

docs/source/en/model_doc/gptsan-japanese.md

@@ -50,7 +50,7 @@ The `generate()` method can be used to generate text using GPTSAN-Japanese model
 >>> model = AutoModel.from_pretrained("Tanrei/GPTSAN-japanese").to(device)
 >>> x_tok = tokenizer("は、", prefix_text="織田信長", return_tensors="pt")
 >>> torch.manual_seed(0)
->>> gen_tok = model.generate(x_tok.input_ids.to(model.device), token_type_ids=x_tok.token_type_ids.to(mdoel.device), max_new_tokens=20)
+>>> gen_tok = model.generate(x_tok.input_ids.to(model.device), token_type_ids=x_tok.token_type_ids.to(model.device), max_new_tokens=20)
 >>> tokenizer.decode(gen_tok[0])
 '織田信長は、2004年に『戦国BASARA』のために、豊臣秀吉'
 ```

docs/source/en/quicktour.md

@@ -124,7 +124,7 @@ Create a [`Pipeline`] object and select a task. By default, [`Pipeline`] downloa
 <hfoptions id="pipeline-tasks">
 <hfoption id="text generation">
 
-Use [`~infer_device`] to automatically detect an available accelerator for inference.
+Use [`infer_device`] to automatically detect an available accelerator for inference.
 
 ```py
 from transformers import pipeline, infer_device
@@ -144,7 +144,7 @@ pipeline("The secret to baking a good cake is ", max_length=50)
 </hfoption>
 <hfoption id="image segmentation">
 
-Use [`~infer_device`] to automatically detect an available accelerator for inference.
+Use [`infer_device`] to automatically detect an available accelerator for inference.
 
 ```py
 from transformers import pipeline, infer_device
@@ -171,7 +171,7 @@ segments[1]["label"]
 </hfoption>
 <hfoption id="automatic speech recognition">
 
-Use [`~infer_device`] to automatically detect an available accelerator for inference.
+Use [`infer_device`] to automatically detect an available accelerator for inference.
 
 ```py
 from transformers import pipeline, infer_device

docs/source/en/serving.md

@@ -21,7 +21,7 @@ Transformer models can be efficiently deployed using libraries such as vLLM, Tex
 > [!TIP]
 > Responses API is now supported as an experimental API! Read more about it [here](#responses-api).
 
-Apart from that you can also serve transformer models easily using the `transformers serve` CLI. This is ideal for experimentation purposes, or to run models locally for personal and private use.
+You can also serve transformer models with the `transformers serve` CLI. With Continuous Batching, `serve` now delivers solid throughput and latency well suited for evaluation, experimentation, and moderate-load local or self-hosted deployments. While vLLM, SGLang, or other inference engines remain our recommendations for large-scale production, `serve` avoids the extra runtime and operational overhead, and is on track to gain more production-oriented features.
 
 In this document, we dive into the different supported endpoints and modalities; we also cover the setup of several user interfaces that can be used on top of `transformers serve` in the following guides:
 - [Jan (text and MCP user interface)](./jan.md)
@@ -58,7 +58,7 @@ or by sending an HTTP request, like we'll see below.
 
 ## Chat Completions - text-based
 
-See below for examples for text-based requests. Both LLMs and VLMs should handle 
+See below for examples for text-based requests. Both LLMs and VLMs should handle
 
 <hfoptions id="chat-completion-http">
 <hfoption id="curl">
@@ -366,6 +366,40 @@ The `transformers serve` server is also an MCP client, so it can interact with M
 
 <!-- TODO: example with a minimal python example, and explain that it is possible to pass a full generation config in the request -->
 
+## Continuous Batching
 
+Continuous Batching (CB) lets the server dynamically group and interleave requests so they can share forward passes on the GPU. Instead of processing each request sequentially, `serve` adds new requests as others progress (prefill) and drops finished ones during decode. The result is significantly higher GPU utilization and better throughput without sacrificing latency for most workloads.
+
+Thanks to this, evaluation, experimentation, and moderate-load local/self-hosted use can now be handled comfortably by `transformers serve` without introducing an extra runtime to operate.
+
+### Enable CB in serve
+
+CB is opt-in and currently applies to chat completions.
+
+```sh
+transformers serve \
+  --continuous-batching
+  --attn_implementation sdpa_paged
+```
+
+
+### Performance tips
+
+- Use an efficient attention backend when available:
+
+```sh
+transformers serve \
+  --continuous_batching \
+  --attn_implementation paged_attention
+```
+
+> [!TIP]
+> If you choose `paged_attention`, you must install `flash-attn` separately: `pip install flash-attn --no-build-isolation`
+
+- `--dtype {bfloat16|float16}` typically improve throughput and memory use vs. `float32`
+
+- `--load_in_4bit`/`--load_in_8bit` can reduce memory footprint for LoRA setups
+
+- `--force-model <repo_id>` avoids per-request model hints and helps produce stable, repeatable runs
 
 

docs/source/en/tasks/keypoint_matching.md

@@ -0,0 +1,129 @@
+<!--Copyright 2025 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# Keypoint matching
+
+Keypoint matching matches different points of interests that belong to same object appearing in two different images. Most modern keypoint matchers take images as input and output the following:
+
+- **Keypoint coordinates (x,y):** one-to-one mapping of pixel coordinates between the first and the second image using two lists. Each keypoint at a given index in the first list is matched to the keypoint at the same index in the second list.
+- **Matching scores:** Scores assigned to the keypoint matches.
+
+In this tutorial, you will extract keypoint matches with the [`EfficientLoFTR`] model trained with the [MatchAnything framework](https://huggingface.co/zju-community/matchanything_eloftr), and refine the matches. This model is only 16M parameters and can be run on a CPU. You will use the [`AutoModelForKeypointMatching`] class.
+
+```python
+from transformers import AutoImageProcessor, AutoModelForKeypointMatching
+import torch
+
+processor = AutoImageProcessor.from_pretrained("zju-community/matchanything_eloftr")
+model = AutoModelForKeypointMatching.from_pretrained("zju-community/matchanything_eloftr"))
+```
+
+Load two images that have the same object of interest. The second photo is taken a second apart, it's colors are edited, and it is further cropped and rotated.
+
+<div style="display: flex; align-items: center;">
+    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg" 
+         alt="Bee" 
+         style="height: 200px; object-fit: contain; margin-right: 10px;">
+    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee_edited.jpg" 
+         alt="Bee edited" 
+         style="height: 200px; object-fit: contain;">
+</div>
+
+```python 
+from transformers.image_utils import load_image
+image1 = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg")
+image2 = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee_edited.jpg")
+
+images = [image1, image2]
+```
+
+We can pass the images to the processor and infer.
+
+```python
+inputs = processor(images, return_tensors="pt")
+with torch.no_grad():
+    outputs = model(**inputs)
+```
+
+We can postprocess the outputs. The threshold parameter is used to refine noise (lower confidence thresholds) in the output matches.
+
+```python
+image_sizes = [[(image.height, image.width) for image in images]]
+
+outputs = processor.post_process_keypoint_matching(outputs, image_sizes, threshold=0.2)
+print(outputs)
+```
+
+Here's the outputs.
+
+```
+[{'keypoints0': tensor([[4514,  550],
+          [4813,  683],
+          [1972, 1547],
+          ...
+          [3916, 3408]], dtype=torch.int32),
+  'keypoints1': tensor([[2280,  463],
+          [2378,  613],
+          [2231,  887],
+          ...
+          [1521, 2560]], dtype=torch.int32),
+  'matching_scores': tensor([0.2189, 0.2073, 0.2414, ...
+    ])}]
+``` 
+
+We have trimmed the output but there's 401 matches!
+
+```python
+len(outputs[0]["keypoints0"])
+# 401
+``` 
+
+We can visualize them using the processor's [`~EfficientLoFTRImageProcessor.visualize_keypoint_matching`] method. 
+
+```python
+plot_images = processor.visualize_keypoint_matching(images, outputs)
+plot_images
+```
+
+![Matched Image](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/matched_bees.png)
+
+Optionally, you can use the [`Pipeline`] API and set the task to `keypoint-matching`. 
+
+```python
+from transformers import pipeline 
+
+image_1 = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
+image_2 = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee_edited.jpg"
+
+pipe = pipeline("keypoint-matching", model="zju-community/matchanything_eloftr")
+pipe([image_1, image_2])
+```
+
+The output looks like following.
+
+```bash
+[{'keypoint_image_0': {'x': 2444, 'y': 2869},
+  'keypoint_image_1': {'x': 837, 'y': 1500},
+  'score': 0.9756593704223633},
+ {'keypoint_image_0': {'x': 1248, 'y': 2819},
+  'keypoint_image_1': {'x': 862, 'y': 866},
+  'score': 0.9735618829727173},
+ {'keypoint_image_0': {'x': 1547, 'y': 3317},
+  'keypoint_image_1': {'x': 1436, 'y': 1500},
+  ...
+ }
+]
+```

docs/source/en/tasks/visual_document_retrieval.md

@@ -79,7 +79,7 @@ Index the images offline, and during inference, return the query text embeddings
 Store the image and image embeddings by writing them to the dataset with [`~datasets.Dataset.map`] as shown below. Add an `embeddings` column that contains the indexed embeddings. ColPali embeddings take up a lot of storage, so remove them from the accelerator and store them in the CPU as NumPy vectors.
 
 ```python
-ds_with_embeddings = dataset.map(lambda example: {'embeddings': model(**processor(images=example["image"]).to(devide), return_tensors="pt").embeddings.to(torch.float32).detach().cpu().numpy()})
+ds_with_embeddings = dataset.map(lambda example: {'embeddings': model(**processor(images=example["image"]).to(device), return_tensors="pt").embeddings.to(torch.float32).detach().cpu().numpy()})
 ```
 
 For online inference, create a function to search the image embeddings in batches and retrieve the k-most relevant images. The function below returns the indices in the dataset and their scores for a given indexed dataset, text embeddings, number of top results, and the batch size.

docs/source/ko/_toctree.yml

@@ -997,6 +997,8 @@
         title: WavLM
       - local: model_doc/whisper
         title: Whisper
+      - local: model_doc/xclip
+        title: xclip
       - local: in_translation
         title: XLS-R
       - local: in_translation

docs/source/ko/model_doc/xclip.md

@@ -0,0 +1,84 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# X-CLIP[[x-clip]]
+
+<div class="flex flex-wrap space-x-1">
+<img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white">
+</div>
+
+## 개요[[overview]]
+
+X-CLIP 모델은 Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, Haibin Ling이 [Expanding Language-Image Pretrained Models for General Video Recognition](https://huggingface.co/papers/2208.02816)에서 제안했습니다.
+X-CLIP은 비디오를 위해 [CLIP](clip)을 최소한으로 확장한 모델입니다. 이 모델은 텍스트 인코더, 교차 프레임 비전 인코더, 다중 프레임 통합 Transformer, 그리고 비디오별 프롬프트 생성기로 구성됩니다.
+
+논문의 초록은 아래와 같습니다.
+
+*대조적 언어-이미지 사전 학습은 웹 스케일 데이터로부터 시각-텍스트 공동 표현을 학습하는 데 큰 성공을 거두었으며, 다양한 이미지 작업에 대해 뛰어난 "제로샷(zero-shot)" 일반화 능력을 보여주었습니다. 그러나 이러한 새로운 언어-이미지 사전 학습 방법을 비디오 도메인으로 효과적으로 확장하는 방법은 아직 해결되지 않은 문제입니다. 본 연구에서는 새로운 모델을 처음부터 사전 학습하는 대신, 사전 학습된 언어-이미지 모델을 비디오 인식에 직접 적용하는 간단하면서도 효과적인 접근 방식을 제시합니다. 더 구체적으로, 시간 차원에서 프레임 간의 장기적인 의존성을 포착하기 위해 프레임 간 정보를 명시적으로 교환하는 교차 프레임 어텐션 메커니즘을 제안합니다. 이러한 모듈은 가벼울 뿐만 아니라, 사전 학습된 언어-이미지 모델에 쉽게 통합될 수 있습니다. 또한, 비디오 콘텐츠 정보를 활용하여 식별력 있는 텍스트 프롬프트를 생성하는 비디오별 프롬프팅 기법을 제안합니다. 광범위한 실험을 통해 우리의 접근 방식이 효과적이며 다양한 비디오 인식 시나리오에 일반화될 수 있음을 입증합니다. 특히, 완전 지도 학습 환경에서 우리 접근 방식은 Kinectics-400에서 87.1%의 top-1 정확도를 달성하면서도 Swin-L 및 ViViT-H에 비해 FLOPs를 12배 적게 사용합니다. 제로샷 실험에서는 두 가지 인기 있는 프로토콜 하에서 top-1 정확도 측면에서 현재 최첨단 방법들을 +7.6% 및 +14.9% 능가합니다. 퓨샷(few-shot) 시나리오에서는 레이블이 지정된 데이터가 극히 제한적일 때 이전 최고 방법들을 +32.1% 및 +23.1% 능가합니다.*
+
+팁:
+
+- X-CLIP의 사용법은 [CLIP](clip)과 동일합니다.
+
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/xclip_architecture.png"
+alt="drawing" width="600"/>
+
+<small> X-CLIP 아키텍처. <a href="https://huggingface.co/papers/2208.02816">원본 논문</a>에서 가져왔습니다. </small>
+
+이 모델은 [nielsr](https://huggingface.co/nielsr)님이 기여했습니다.
+원본 코드는 [여기](https://github.com/microsoft/VideoX/tree/master/X-CLIP)에서 찾을 수 있습니다.
+
+## 리소스[[resources]]
+
+X-CLIP을 시작하는 데 도움이 되는 공식 Hugging Face 및 커뮤니티(🌎로 표시) 리소스 목록입니다.
+
+- X-CLIP 데모 노트북은 [여기](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/X-CLIP)에서 찾을 수 있습니다.
+
+여기에 포함할 리소스를 제출하는 데 관심이 있다면, 언제든지 Pull Request를 열어주세요. 검토 후 반영하겠습니다! 리소스는 기존 리소스를 복제하는 대신 새로운 것을 보여주는 것이 이상적입니다.
+
+## XCLIPProcessor[[xclipprocessor]]
+
+[[autodoc]] XCLIPProcessor
+
+## XCLIPConfig[[xclipconfig]]
+
+[[autodoc]] XCLIPConfig
+    - from_text_vision_configs
+
+## XCLIPTextConfig[[xcliptextconfig]]
+
+[[autodoc]] XCLIPTextConfig
+
+## XCLIPVisionConfig[[xclipvisionconfig]]
+
+[[autodoc]] XCLIPVisionConfig
+
+## XCLIPModel[[xclipmodel]]
+
+[[autodoc]] XCLIPModel
+    - forward
+    - get_text_features
+    - get_video_features
+
+## XCLIPTextModel[[xcliptextmodel]]
+
+[[autodoc]] XCLIPTextModel
+    - forward
+
+## XCLIPVisionModel[[xclipvisionmodel]]
+
+[[autodoc]] XCLIPVisionModel
+    - forward