Mull-Tokens: 通用模态的潜在思考
Mull-Tokens: Modality-Agnostic Latent Thinking
AI总结：
本文提出Mull-Tokens，一种无需模态切换的潜在令牌，用于空间、时间等多模态推理，通过预训练和微调在四个挑战性基准上实现3%-16%的提升。
摘要：
推理超越语言；现实世界需要对空间、时间、可能性等进行推理，而语言无法传达。现有多模态模型在图像推理中表现脆弱且难以扩展，依赖调用专业工具、生成图像或手工制作数据切换文本和图像思维。本文提出Mull-Tokens，一种模态无关的潜在令牌，预训练以在图像或文本模态中持有中间信息，使模型能自由思考以获得正确答案。我们研究了受潜在推理框架启发的最佳实践。首先通过交错文本-图像轨迹监督训练Mull-Tokens，然后通过仅使用最终答案进行微调。在四个具有挑战性的空间推理基准上，包括解决谜题和不同视角的任务，我们证明Mull-Tokens在文本-only推理或交错图像-文本推理基础上，平均提升3%，在谜题解决任务中达到最高16%的提升。Mull-Tokens为文本和视觉推理的挑战提供了简单的解决方案，实现多模态抽象思考。
英文摘要：
Reasoning goes beyond language; the real world requires reasoning about space, time, affordances, and much more that words alone cannot convey. Existing multimodal models exploring the potential of reasoning with images are brittle and do not scale. They rely on calling specialist tools, costly generation of images, or handcrafted reasoning data to switch between text and image thoughts. Instead, we offer a simpler alternative -- Mull-Tokens -- modality-agnostic latent tokens pre-trained to hold intermediate information in either image or text modalities to let the model think free-form towards the correct answer. We investigate best practices to train Mull-Tokens inspired by latent reasoning frameworks. We first train Mull-Tokens using supervision from interleaved text-image traces, and then fine-tune without any supervision by only using the final answers. Across four challenging spatial reasoning benchmarks involving tasks such as solving puzzles and taking different perspectives, we demonstrate that Mull-Tokens improve upon several baselines utilizing text-only reasoning or interleaved image-text reasoning, achieving a +3% average improvement and up to +16% on a puzzle solving reasoning-heavy split compared to our strongest baseline. Adding to conversations around challenges in grounding textual and visual reasoning, Mull-Tokens offers a simple solution to abstractly think in multiple modalities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
