请问使用tokenizer/processor制作图文输入的区别

[tettethu]

我是一个刚接触这个领域的新人，遇到了一个很奇怪的问题：

当我试图分别使用AutoProcessor和AutoTokenizer处理输入图文输入（即下方两个函数所示），并喂给Qwen2.5-VL-3B-Instruct获取回答时，所得到的结果有很大不同，但是都明显具备图片感知能力，也就是说图片信息确实传给模型了，那么是什么导致了这种不同呢？

def inference(model,processor,image_path, prompt, sys_prompt="You are a helpful assistant.", max_new_tokens=4096, return_input=False,show_input_text=True):
    '''
    processor,image_path,prompt->qwen2.5-vl response text
    - return_input: if True, return the input tensor as well
    '''
    image = Image.open(image_path)
    image_local_path = "file://" + image_path
    messages = [
        {"role": "system", "content": sys_prompt},
        {"role": "user", "content": [
                {"type": "text", "text": prompt},
                {"image": image_path},
            ]
        },
    ]
    text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    if show_input_text:
        print("input text:", text)
    # image_inputs, video_inputs = process_vision_info([messages])
    inputs = processor(text=[text], images=[image], padding=True, return_tensors="pt")
    inputs = inputs.to(model.device)

    output_ids = model.generate(**inputs, max_new_tokens=max_new_tokens)
    generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
    output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
    if return_input:
        return output_text[0], inputs
    else:
        return output_text[0]

def inference_with_tokenizer(model, tokenizer, image_path, prompt, sys_prompt="You are a helpful assistant.", max_new_tokens=4096, return_input=False, show_input_text=True):
    '''
    processor,image_path,prompt->qwen2.5-vl response text
    - return_input: if True, return the input tensor as well
    '''
    text=" <|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n{}<|vision_start|><img>{}</img><|vision_end|><|im_end|>\n<|im_start|>assistant".format(prompt,image_path)
    inputs=tokenizer(text, return_tensors="pt", padding=True)
    print(type(inputs))
    inputs=inputs.to(model.device)
    output_ids=model.generate(**inputs, max_new_tokens=max_new_tokens)
    response=ids2response(inputs.input_ids, output_ids, tokenizer)
    if return_input:
        return response, inputs
    else:
        return response

两者除了一个按照使用说明分别将文本和图片传给processor得到输入，一个将按照规则排列好的字符串中的<|image_pad|>改为图片路径然后传给tokenizer得到输入，没有其他区别，为什么结果会有很大不同，我一开始猜测tokenizer可能现在无法处理这种输入了，但是实际上输出内容还是高度和图片相关的，所以很好奇是什么导致了这种不同

作为参考，以下是输入数据：
prompt= "Describe this image."
image:

两个函数的输出分别为：
使用processor:
[processor]qwen2.5-vl says:The image shows the Microsoft Outlook app page on an Android device's Google Play Store. The app is developed by Microsoft Corporation and has a 4.5-star rating based on 8 million reviews. It has been downloaded over 500 million times and is available for everyone. The app allows users to manage their email and calendars, collaborate with anyone, and anywhere. Below the main app listing, there are other Microsoft apps listed: Microsoft OneDrive (4.6 stars), Microsoft Teams (4.7 stars), Microsoft Authenticator (installed), and Microsoft To Do: Lists & Tasks (4.5 stars).

使用tokenizer
[tokenzer]qwen2.5-vl says:: The image appears to be a screenshot of an email client interface, specifically Outlook. The layout includes several elements typical of an email application:

1. Top Bar: This bar contains the Outlook logo on the left and the search bar in the center. Below the search bar, there is a navigation menu with options like "Home," "Inbox," "Sent Items," "Trash," "Categories," and "Settings."

2. Main Content Area: This section displays the inbox view, showing various emails from different senders. Each email has a preview of the subject line and a small thumbnail image on the right side.

3. Sidebar: On the right side, there is a sidebar that likely contains additional options such as "Categories," "Labels," and possibly a search function.

4. Email Details: When you click on an email, it opens in a new window or tab, displaying more detailed information about the email, including the sender's name, the date and time it was sent, the subject, and the body text.

5. Buttons and Icons: At the bottom of the main content area, there are buttons for actions such as "Reply," "Forward," "Delete," and "More Actions" (which might include options like "Mark as Read," "Flag," etc.).

Overall, the interface is clean and organized, designed to facilitate easy navigation and management of emails.

[tettethu]

对于这个问题，又发现了一些新的特性：

并非所有图片经过tokenizer处理模型都能感知到，一些原来可能传给过Qwen-VL的tokenizer的图片或者曾经被用来做过微调的图片（文件路径改动过）似乎能被感知到，而我随便截了张图通过tokenizer传给模型似乎感知不到（模型输出“I'm sorry, but I'm unable to view or access images directly. Could you please describe the image in words?“

所以，Qwen2.5-VL确实不再能使用tokenizer传入图片，但是到底为什么此前将文件路径直接传给tokenizer模型也存在看得到的情况呢，非常好奇，求了解的大佬解惑