视觉草图板：作为多模态语言模型视觉思维链的草图绘制
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models
AI总结：
提出Sketchpad框架，赋予多模态语言模型视觉草图板，通过绘制线条、框等辅助推理，在数学和视觉任务上显著提升性能，并创下多项新纪录。
摘要：
人类通过绘图来促进推理：在解决几何问题时，我们会画辅助线；在地图上推理时，我们会做标记和圈画；我们用草图来强化想法并缓解有限容量的工作记忆。然而，当前的多模态语言模型（LMs）缺乏此类行为。现有的思维链和工具使用范式仅使用文本作为中间推理步骤。在这项工作中，我们引入了Sketchpad，一个为多模态语言模型提供视觉草图板及在其上绘图工具的框架。语言模型根据其绘制的视觉产物进行规划和推理。与先前使用文本到图像模型使语言模型能够绘图的工作不同，Sketchpad使语言模型能够使用线条、框、标记等进行绘图，这更接近人类的草图绘制，且更有利于推理。Sketchpad还可以在草图绘制过程中使用专门的视觉模型（例如，使用目标检测模型绘制边界框，使用分割模型绘制掩码），以进一步增强视觉感知和推理。我们在广泛的数学任务（包括几何、函数、图形和国际象棋）以及复杂的视觉推理任务上进行了实验。与不使用草图绘制的强基座模型相比，Sketchpad在所有任务上均显著提升了性能，在数学任务上平均提升12.7%，在视觉任务上平均提升8.6%。配备Sketchpad的GPT-4o在所有任务上创下了新的最先进水平，包括V*Bench（80.3%）、BLINK空间推理（83.9%）和视觉对应（80.8%）。所有代码和数据见https://visualsketchpad.github.io/。
英文摘要：
Humans draw to facilitate reasoning: we draw auxiliary lines when solving geometry problems; we mark and circle when reasoning on maps; we use sketches to amplify our ideas and relieve our limited-capacity working memory. However, such actions are missing in current multimodal language models (LMs). Current chain-of-thought and tool-use paradigms only use text as intermediate reasoning steps. In this work, we introduce Sketchpad, a framework that gives multimodal LMs a visual sketchpad and tools to draw on the sketchpad. The LM conducts planning and reasoning according to the visual artifacts it has drawn. Different from prior work, which uses text-to-image models to enable LMs to draw, Sketchpad enables LMs to draw with lines, boxes, marks, etc., which is closer to human sketching and better facilitates reasoning. Sketchpad can also use specialist vision models during the sketching process (e.g., draw bounding boxes with object detection models, draw masks with segmentation models), to further enhance visual perception and reasoning. We experiment with a wide range of math tasks (including geometry, functions, graphs, and chess) and complex visual reasoning tasks. Sketchpad substantially improves performance on all tasks over strong base models with no sketching, yielding an average gain of 12.7% on math tasks, and 8.6% on vision tasks. GPT-4o with Sketchpad sets a new state of the art on all tasks, including V*Bench (80.3%), BLINK spatial reasoning (83.9%), and visual correspondence (80.8%). All codes and data are in https://visualsketchpad.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
