arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.15460cs.LGcs.CV
收藏

在简单视觉规划任务中多模态大语言模型推理的分布外泛化

On the Out-of-Distribution Generalization of Reasoning in Multimodal LLMs for Simple Visual Planning Tasks

Tübingen AI Center -- University of Tübingen(图宾根人工智能中心 -- 图宾根大学);EPFL(瑞士联邦理工学院);ELLIS Institute Finland -- Aalto University(芬兰ELLIS研究所 -- 阿尔托大学)

Yannic Neuhaus, Nicolas Flammarion, Matthias Hein, Francesco Croce

AI总结:

研究多模态大语言模型在简单视觉规划任务中推理能力的分布外泛化,发现结合多种文本格式的推理方法效果最佳,纯文本模型表现优于图像输入模型。

摘要:

将推理能力整合到大型语言模型和大型视觉-语言模型中,最近显著提升了它们的能力。然而,推理模型的泛化能力仍然定义模糊且理解不足。在本工作中,我们提出一个评估框架,严格检验链式推理(CoT)方法在简单规划任务中的泛化能力。具体而言,我们考虑一个基于网格的导航任务,在此任务中,模型会得到一张地图,并必须输出一系列移动,引导玩家从起点位置移动到目标位置,同时避开障碍物。该任务的多样性和数据允许我们使用不同的输入表示(视觉和文本)和Co T推理策略对模型变体进行微调,并在分布内(ID)和分布外(OOD)测试条件下系统评估它们。我们的实验表明,尽管Co T推理在所有表示上提高了分布内泛化能力,但分布外泛化(例如,到更大的地图)在大多数情况下仍非常有限,尤其是在控制与ID数据的简单匹配时。令人惊讶的是,我们发现结合多种文本格式的推理轨迹产生了最佳(且非平凡)的OOD泛化能力。最后,纯文本模型在使用图像输入的模型上始终表现更优,包括最近提出的一种依赖于潜在空间推理的方法。

英文摘要:

Integrating reasoning in large language models and large vision-language models has recently led to significant improvement of their capabilities. However, the generalization of reasoning models is still vaguely defined and poorly understood. In this work, we present an evaluation framework to rigorously examine how well chain-of-thought (CoT) approaches generalize on a simple planning task. Specifically, we consider a grid-based navigation task in which a model is provided with a map and must output a sequence of moves that guides a player from a start position to a goal while avoiding obstacles. The versatility of the task and its data allows us to fine-tune model variants using different input representations (visual and textual) and CoT reasoning strategies, and systematically evaluate them under both in-distribution (ID) and out-of-distribution (OOD) test conditions. Our experiments show that, while CoT reasoning improves in-distribution generalization across all representations, out-of-distribution generalization (e.g., to larger maps) remains very limited in most cases when controlling for trivial matches with the ID data. Surprisingly, we find that reasoning traces which combine multiple text formats yield the best (and non-trivial) OOD generalization. Finally, purely text-based models consistently outperform those utilizing image-based inputs, including a recently proposed approach relying on latent space reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑