KeyError: 'type' in PixelReasoner-SFT-Data

[zhang-haojie]

Hi, I encountered an error when running the instruction tuning pipeline with Qwen-VL, specifically during vision input processing. The stack trace is as follows:

File "/usr/local/lib/python3.12/dist-packages/qwen_vl_utils/vision_process.py", line 346, in extract_vision_info
    or ele["type"] in ("image", "image_url", "video")
KeyError: 'type'

However, the sample from TIGER-Lab/PixelReasoner-SFT-Data/release.json looks like this:

ipdb> messages[0][0]
{'role': 'system', 'content': [{'text': 'You are a helpful assistant.\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{"type": "function", "function": {"name": "crop_image_normalized", "description": "Zoom in on the image based on the bounding box coordinates.", "parameters": {"type": "object", "properties": {"bbox_2d": {"type": "array", "description": "normalized coordinates for bounding box of the region you want to zoom in. Values should be within [0.0,1.0].", "items": {"type": "number"}}, "target_image": {"type": "number", "description": "The index of the image to crop. Index from 1 to the number of images. Choose 1 to operate on original image."}}, "required": ["bbox_2d", "target_image"]}}}\n{"type": "function", "function": {"name": "select_frames", "description": "Select frames from a video.", "parameters": {"type": "object", "properties": {"target_frames": {"type": "array", "description": "List of frame indices to select from the video (no more than 8 frames in total).", "items": {"type": "integer", "description": "Frame index from 1 to 16."}}}, "required": ["target_frames"]}}}\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{"name": <function-name>, "arguments": <args-json-object>}\n</tool_call>'}]}

As you can see, the inner content dictionary does not have a "type" key (just "text").

{
  "role": "system",
  "content": [
    {
      "text": "You are a helpful assistant.\n\n# Tools\n..."
    }
  ]
}

[cjakfskvnad]

I think this is related to qwen_vl_utils version. Try use qwen_vl_utils==0.0.14