Learning to See Before Seeing：揭秘语言预训练中LLM的视觉先验
Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training
AI总结：
本研究揭示LLM仅靠文本预训练形成的视觉先验由可分离的感知与推理先验构成，提出以数据为核心的视觉感知LLM预训练方案，经大规模实验验证，为多模态LLM发展提供新思路。
摘要：
大语言模型（LLM）尽管仅在文本上训练，却出人意料地发展出了丰富的视觉先验。这些先验使得仅需相对少量的多模态数据，就能解锁潜在的视觉能力以完成视觉任务，在某些情况下，甚至从未见过图像也能执行视觉任务。通过系统分析，我们揭示出视觉先验——即在语言预训练期间获得的关于视觉世界的隐性、涌现性知识——由可分离的感知先验和推理先验组成，二者具有独特的缩放趋势与起源。我们表明，LLM的潜在视觉推理能力主要通过在以推理为中心的数据（如代码、数学、学术内容）上进行预训练而发展，且呈逐步缩放态势。这种从语言预训练中获得的推理先验具有可迁移性，普遍适用于视觉推理任务。相比之下，感知先验更广泛地从海量语料中涌现，且感知能力对视觉编码器和视觉指令微调数据更为敏感。同时，描述视觉世界的文本被证明至关重要，但其对性能的提升作用会迅速饱和。基于这些洞见，我们提出了一种以数据为核心的预训练具备视觉感知能力LLM的方案，并在1T token规模的预训练中对其进行了验证。我们的发现基于超过100组对照实验，消耗50万GPU小时，覆盖多模态大语言模型（MLLM）的完整构建流程——从LLM预训练到视觉对齐与有监督多模态微调——涉及五种模型规模、广泛的数据类别与混合方式，以及多种适配设置。除主要发现外，我们还提出并研究了若干假设，并引入了多级存在基准（Multi-Level Existence Bench, MLE-Bench）。总体而言，这项工作提供了一种从语言预训练中有意识地培育视觉先验的新途径，为下一代多模态LLM铺平了道路。
英文摘要：
Large Language Models (LLMs), despite being trained on text alone, surprisingly develop rich visual priors. These priors allow latent visual capabilities to be unlocked for vision tasks with a relatively small amount of multimodal data, and in some cases, to perform visual tasks without ever having seen an image. Through systematic analysis, we reveal that visual priors-the implicit, emergent knowledge about the visual world acquired during language pre-training-are composed of separable perception and reasoning priors with unique scaling trends and origins. We show that an LLM's latent visual reasoning ability is predominantly developed by pre-training on reasoning-centric data (e.g., code, math, academia) and scales progressively. This reasoning prior acquired from language pre-training is transferable and universally applicable to visual reasoning. In contrast, a perception prior emerges more diffusely from broad corpora, and perception ability is more sensitive to the vision encoder and visual instruction tuning data. In parallel, text describing the visual world proves crucial, though its performance impact saturates rapidly. Leveraging these insights, we propose a data-centric recipe for pre-training vision-aware LLMs and verify it in 1T token scale pre-training. Our findings are grounded in over 100 controlled experiments consuming 500,000 GPU-hours, spanning the full MLLM construction pipeline-from LLM pre-training to visual alignment and supervised multimodal fine-tuning-across five model scales, a wide range of data categories and mixtures, and multiple adaptation setups. Along with our main findings, we propose and investigate several hypotheses, and introduce the Multi-Level Existence Bench (MLE-Bench). Together, this work provides a new way of deliberately cultivating visual priors from language pre-training, paving the way for the next generation of multimodal LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
