Whiteboard-of-Thought：跨模态逐步思考
Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities
AI总结：
针对多模态大语言模型难以通过视觉推理解答文本查询的问题，提出Whiteboard-of-Thought提示方法，借模型代码绘图能力解锁跨模态视觉推理，在多项相关任务上取得最优结果。
摘要：
当遇到涉及视觉思考的问题时，人类会自然地切换推理模态，通常会形成心理意象或绘制视觉辅助工具。大语言模型通过将中间推理以文本形式表达为思维链，已在算术和符号推理中展现出可观的成果，但即便经过大规模多模态预训练，这类模型仍难以将该能力拓展至解答可通过视觉推理轻松解决的文本查询问题。我们提出了一种简单的方法——Whiteboard-of-Thought（思维白板）提示，以解锁多模态大语言模型的跨模态视觉推理能力。Whiteboard-of-Thought提示为多模态大语言模型提供一块隐喻意义上的“白板”，将推理步骤绘制成图像，再将这些图像传回模型做进一步处理。我们发现，无需示例演示或专用模块，仅利用模型已有的、借助Matplotlib和Turtle等库编写代码的能力即可实现该方法。这一简单方法在四项涉及视觉与空间推理的高难度自然语言任务上取得了当前最优结果。我们发现，采用思维链的GPT-4o在多种场景下表现严重失效，其中不止一种场景的准确率为0%，而在相同场景下，Whiteboard-of-Thought可实现高达92%的准确率。我们还详细探究了该技术的适用场景及其误差来源。
英文摘要：
When presented with questions involving visual thinking, humans naturally switch reasoning modalities, often forming mental images or drawing visual aids. Large language models have shown promising results in arithmetic and symbolic reasoning by expressing intermediate reasoning in text as a chain of thought, yet struggle to extend this capability to answer text queries that are easily solved by visual reasoning, even with extensive multimodal pretraining. We introduce a simple method, whiteboard-of-thought prompting, to unlock the visual reasoning capabilities of multimodal large language models across modalities. Whiteboard-of-thought prompting provides multimodal large language models with a metaphorical `whiteboard' to draw out reasoning steps as images, then returns these images back to the model for further processing. We find this can be accomplished with no demonstrations or specialized modules, instead leveraging models' existing ability to write code with libraries such as Matplotlib and Turtle. This simple approach shows state-of-the-art results on four difficult natural language tasks that involve visual and spatial reasoning. We identify multiple settings where GPT-4o using chain-of-thought fails dramatically, including more than one where it achieves $0\%$ accuracy, while whiteboard-of-thought enables up to $92\%$ accuracy in these same settings. We present a detailed exploration of where the technique succeeds as well as its sources of error.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
