DeepSketcher：内化视觉操作以实现多模态推理
DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoning
AI总结：
DeepSketcher通过构建图像-文本交织数据集和自包含模型，实现了无需外部工具的视觉思考，提升了多模态推理能力。
摘要：
'图像思考'范式标志着视觉语言模型（VLMs）推理方式的重大转变，从文本主导的推理链转向图像交互推理。通过调用视觉工具或生成中间视觉表示，VLMs可以迭代关注细节区域，实现更深入的图像理解和更忠实的多模态推理。然而，该范式在数据构建准确性、结构设计和更广泛的应用场景上仍有探索空间。为此，我们提出了DeepSketcher，包含一个图像-文本交织的数据集和一个自包含模型。数据集包含31k条推理轨迹，涵盖多样化的工具调用和编辑图像，覆盖广泛的数据类型和操作指令，具有高注释准确性。基于此资源，我们设计了一个能够执行交织图像-文本推理并原生生成'视觉思考'的模型，直接在视觉嵌入空间中操作，而非调用外部工具和反复重新编码生成的图像。这种设计实现了无需工具的更灵活的'图像思考'。在多模态推理基准上的广泛实验验证了模型的性能，证明了数据集的实用性和模型设计的有效性。
英文摘要：
The "thinking with images" paradigm represents a pivotal shift in the reasoning of Vision Language Models (VLMs), moving from text-dominant chain-of-thought to image-interactive reasoning. By invoking visual tools or generating intermediate visual representations, VLMs can iteratively attend to fine-grained regions, enabling deeper image understanding and more faithful multimodal reasoning. As an emerging paradigm, however, it still leaves substantial room for exploration in data construction accuracy, structural design, and broader application scenarios, which offer rich opportunities for advancing multimodal reasoning. To further advance this line of work, we present DeepSketcher, a comprehensive suite comprising both an image-text interleaved dataset and a self-contained model. The dataset contains 31k chain-of-thought (CoT) reasoning trajectories with diverse tool calls and resulting edited images, covering a wide range of data types and manipulation instructions with high annotation accuracy. Building on this resource, we design a model that performs interleaved image-text reasoning and natively generates "visual thoughts" by operating directly in the visual embedding space, rather than invoking external tools and repeatedly re-encoding generated images. This design enables tool-free and more flexible "thinking with images". Extensive experiments on multimodal reasoning benchmarks demonstrate strong performance, validating both the utility of the dataset and the effectiveness of the model design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
