模仿还是推理：重新思考视觉-语言模型中的多模态上下文学习
Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
AI总结：
本文通过分布偏移评估和带推理示例的MM-ICL流程，揭示VLM多模态上下文学习更像复制而非真正利用示例推理。
摘要：
视觉-语言模型（VLM）被广泛认为表现出上下文学习（ICL），这一特性与其仅语言版本类似。尽管近期研究表明VLM能够执行多模态ICL（MM-ICL），但研究显示它们往往依赖浅层启发式方法——例如复制或多数投票——而非真正的任务理解。我们通过在分布偏移下评估VLM来重新审视这一假设，在这种情况下，支持示例来自与查询不同的数据集。令人惊讶的是，随着示例增多，性能往往下降，而且模型倾向于复制答案，而不是从中学习。为进一步研究，我们提出了一种新的带推理的MM-ICL流程，该流程在每个示例中除答案外还增强生成的理由。我们在需要感知和推理的数据集上，对从3B到72B的开源VLM以及Gemini 2.0等专有模型进行了广泛而全面的实验。我们通过改变示例数量、检索方法、理由质量和分布进行了受控研究。结果表明，性能对这些因素的敏感性有限，说明当前VLM并未按MM-ICL的预期有效利用示例级信息。
英文摘要：
Vision-language models (VLMs) are widely assumed to exhibit in-context learning (ICL), a property similar to that of their language-only counterparts. While recent work suggests VLMs can perform multimodal ICL (MM-ICL), studies show they often rely on shallow heuristics -- such as copying or majority voting -- rather than true task understanding. We revisit this assumption by evaluating VLMs under distribution shifts, where support examples come from a dataset different from the query. Surprisingly, performance often degrades with more demonstrations, and models tend to copy answers rather than learn from them. To investigate further, we propose a new MM-ICL with Reasoning pipeline that augments each demonstration with a generated rationale alongside the answer. We conduct extensive and comprehensive experiments on both perception- and reasoning-required datasets with open-source VLMs ranging from 3B to 72B and proprietary models such as Gemini 2.0. We conduct controlled studies varying shot count, retrieval method, rationale quality, and distribution. Our results show limited performance sensitivity across these factors, suggesting that current VLMs do not effectively utilize demonstration-level information as intended in MM-ICL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
