arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.04411cs.CLcs.AIcs.CV
收藏

响应未能揭示理解:揭示视觉文档理解中内部表征与响应之间的差距

Responses Fall Short of Understanding: Revealing the Gap between Internal Representations and Responses in Visual Document Understanding

Human Informatics Labs., NTT, Inc.(日本电信电话株式会社人类信息学实验室)

Haruka Kawasaki, Ryota Tanaka, Kyosuke Nishida

AI总结:

本文研究了视觉文档理解中内部表征与生成响应之间的差距,发现任务所需信息在中间层更线性编码,并通过微调中间层提升性能。

摘要:

视觉文档理解(VDU)是大型视觉语言模型(LVLMs)面临的具有挑战性的任务,需要整合视觉感知、文本识别和结构布局的推理。尽管最近的LVLMs在VDU基准上有所进展,但其性能通常基于生成的响应进行评估,这可能无法反映模型是否实际捕获了所需信息。本文通过线性探测研究了不同层的LLM中解决VDU任务所需信息的表示。研究发现(1)内部表征与生成响应之间存在明显差距,(2)任务所需信息通常在中间层比最终层更线性编码。受这些发现启发,我们探索了针对中间层的微调策略。实验表明,微调中间层可以同时提高线性探测准确率和响应准确率,缩小差距。

英文摘要:

Visual document understanding (VDU) is a challenging task for large vision language models (LVLMs), requiring the integration of visual perception, text recognition, and reasoning over structured layouts. Although recent LVLMs have shown progress on VDU benchmarks, their performance is typically evaluated based on generated responses, which may not necessarily reflect whether the model has actually captured the required information internally. In this paper, we investigate how information required to solve VDU tasks is represented across different layers of LLMs within LVLMs using linear probing. Our study reveals that (1) there is a clear gap between internal representations and generated responses, and (2) information required to solve the task is often encoded more linearly from intermediate layers than from the final layer. Motivated by these findings, we explore fine-tuning strategies that target intermediate layers. Experiments show that fine-tuning intermediate layers improves both linear probing accuracy and response accuracy while narrowing the gap.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑