视觉成为文本:在视觉-语言模型中定位OCR路由瓶颈
Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models
AI总结:
研究揭示了视觉-语言模型中OCR信息进入语言处理流的瓶颈,通过因果干预分析三种架构,发现不同架构对场景文本和单阶段投影模型的敏感层不同,且PCA方向在不同数据集间可迁移,模块化OCR电路模型中移除OCR可提升计数性能。
摘要:
视觉-语言模型(VLMs)能够从图像中读取文本,但OCR信息如何进入语言处理流?本文通过因果干预分析三种架构(Qwen3-VL、Phi-4、InternVL3.5),计算原始图像与文本修补版本的激活差异,识别出架构特定的OCR瓶颈。深度栈模型(Qwen)在中层(约50%)对场景文本最敏感,而单阶段投影模型(Phi-4、InternVL)在早期层(6-25%)最敏感,但最大效果层因数据集而异。OCR信号显著低维:PC1可解释72.9%的方差。关键发现是PCA方向在不同数据集间可迁移,表明共享的文本处理路径。令人惊讶的是,在具有模块化OCR电路的模型(如Qwen3-VL-4B)中,移除OCR可提升计数性能(最高+6.9个百分点),表明OCR在足够模块化的架构中干扰其他视觉处理。
英文摘要:
Vision-language models (VLMs) can read text from images, but where does this optical character recognition (OCR) information enter the language processing stream? We investigate the OCR routing mechanism across three architecture families (Qwen3-VL, Phi-4, InternVL3.5) using causal interventions. By computing activation differences between original images and text-inpainted versions, we identify architecture-specific OCR bottlenecks whose dominant location depends on the vision-language integration strategy: DeepStack models (Qwen) show peak sensitivity at mid-depth (about 50%) for scene text, while single-stage projection models (Phi-4, InternVL) peak at early layers (6-25%), though the exact layer of maximum effect varies across datasets. The OCR signal is remarkably low-dimensional: PC1 captures up to 72.9% of variance. Crucially, principal component analysis (PCA) directions learned on one dataset transfer to others, demonstrating shared text-processing pathways. Surprisingly, in models with modular OCR circuits (notably Qwen3-VL-4B), OCR removal can improve counting performance (up to +6.9 percentage points), suggesting OCR interferes with other visual processing in sufficiently modular architectures.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。