是什么在阻碍潜在视觉推理？
What's Holding Back Latent Visual Reasoning?
AI总结：
本研究探讨了现有模型如何利用潜在令牌，发现潜在令牌在最终预测中起作用有限，主要问题在于训练数据中潜在令牌信息有限且推理时生成的潜在令牌偏离真实表示，需要高质量数据和更精确的潜在令牌预测来推动发展。
摘要：
人类通过心理模拟中间视觉步骤来解决复杂视觉问题，而非仅通过语言推理。受此启发，近期有关视觉-语言模型的工作探索了连续潜在令牌作为中间视觉想象步骤的链式推理。在本工作中，我们研究了近期模型如何利用此类潜在令牌。令人惊讶的是，当潜在令牌被无信息的占位符令牌替代时，模型准确性不受影响。这表明潜在令牌在模型最终预测中起最小的因果作用。为了更好地理解这一现象，我们分析了由oracle潜在表示提供的训练信号以及推理时生成的潜在令牌质量。我们的实验揭示了两个阻碍潜在视觉推理的关键问题：首先，在大多数现有数据集中，oracle潜在令牌提供的信息有限，仅超出原始图像，且不显著简化任务，导致模型在训练时忽略它们，并在推理时有效绕过它们。当在诊断数据集上微调时，其中潜在令牌为最终预测提供充分支持，我们显示模型可以因果依赖于它们。其次，在推理时生成的潜在令牌偏离其对应的oracle表示，坍缩到狭窄区域，即使模型依赖它们也无法获得收益。总体而言，我们的发现表明，未来潜在视觉推理的进步取决于两个关键支柱：具有信息性中间步骤的高质量数据集和更精确的潜在令牌预测。
英文摘要：
Humans can approach complex visual problems by mentally simulating intermediate visual steps, rather than reasoning through language alone. Inspired by this, several works on Vision-Language Models have recently explored chain-of-thought reasoning with continuous latent tokens as intermediate visual imagination steps. In this work, we investigate how recent models leverage such latent tokens. Surprisingly, we find that model accuracy is unaffected when latent tokens are replaced by uninformative dummy tokens. This indicates that latent tokens play a minimal causal role in the model's final prediction. To better understand this phenomenon, we analyze both the training signal provided by oracle latent representations and the quality of the latent tokens generated at inference time. Our experiments reveal two crucial issues holding back latent visual reasoning: First, in most existing datasets, oracle latent tokens provide limited additional information beyond the original image and do not substantially simplify the task, leading models to ignore them during training and effectively bypassing them at inference time. When fine-tuned on a diagnostic dataset, in which latent tokens provide sufficient support for the final prediction, we show that models can causally rely on them. Second, the latent tokens produced at inference time deviate from their corresponding oracle representations, collapsing to a narrow region and preventing benefits even when the model relies on them. Overall, our findings suggest that future progress in latent visual reasoning depends on two key pillars: high-quality datasets with informative intermediate steps and more precise latent token prediction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
