Visual document understanding (VDU) is a challenging task for large vision language models (LVLMs), requiring the integration of visual perception, text recognition, and reasoning over structured layouts. Although recent LVLMs have shown progress on VDU benchmarks, their performance is typically evaluated based on generated responses, which may not necessarily reflect whether the model has actually captured the required information internally. In this paper, we investigate how information required to solve VDU tasks is represented across different layers of LLMs within LVLMs using linear probing. Our study reveals that (1) there is a clear gap between internal representations and generated responses, and (2) information required to solve the task is often encoded more linearly from intermediate layers than from the final layer. Motivated by these findings, we explore fine-tuning strategies that target intermediate layers. Experiments show that fine-tuning intermediate layers improves both linear probing accuracy and response accuracy while narrowing the gap.
LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating
MAIS, Institute of Automation, Chinese Academy of Sciences(人工智能研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);Taobao & Tmall Group of Alibaba(阿里巴巴淘宝与天猫集团);Zhongguancun Academy, Beijing(中关村学院,北京)
Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?
Northeastern University, Boston, MA, USA(东北大学,波士顿,马萨诸塞州,美国);The University of Texas MD Anderson Cancer Center, Houston, Texas, USA(德克萨斯大学MD安德森癌症中心,休斯顿,德克萨斯州,美国);King’s College London, London, UK(伦敦国王学院,伦敦,英国);The University of Texas at Austin, Austin, Texas, USA(德克萨斯大学奥斯汀分校,奥斯汀,德克萨斯州,美国)
Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding
CUHK MMLab(香港中文大学MML实验室);vivo AI Lab(vivo人工智能实验室);CPII under InnoHK(创新工场下的CPII);Shanghai AI Lab & Shenzhen Institute of Advanced Technology, CAS(上海人工智能实验室及深圳先进技术研究所,中国科学院)
Tabular PDF Information Extraction with Local LLMs and Layout-Aware Parsing: A Reliability Evaluation
Faculty of Engineering, Universitas Swadaya Gunung Jati(工程学院,Swadaya Gunung Jati大学);University of California, Irvine(加州大学伊维奇分校);UNIR, La Rioja(UNIR,拉里奥ja);Universitas Diponegoro(迪波内戈罗大学)
Split Personality Training: Revealing Latent Knowledge Through Alternate Personalities
University of Bergen, Bergen, Norway(卑尔根大学);Independent Researcher, London, UK(独立研究者);University College London, London, UK(伦敦大学学院);Independent Researcher, Berlin, Germany(独立研究者);Independent Researcher(独立研究者);Head of Research(研发负责人)
Decisive: Guiding User Decisions with Optimal Preference Elicitation from Unstructured Documents
Adobe Research, India(Adobe印度研究实验室);IIT Madras(印度理工学院马德拉斯分校);IIT Roorkee(印度理工学院罗尔基分校);IIIT Hyderabad(Hyderabad印度理工学院);University of Maryland, College Park(马里兰大学学院公园分校)
Unveiling the Deficiencies of Pre-trained Text-and-Layout Models in Real-world Visually-rich Document Information Extraction
School of Computer Science, Fudan University(复旦大学计算机科学学院);School of Statistics and Data Science, Nankai University(南开大学统计与数据科学学院);Ant Tiansuan Security Lab, Ant Group(蚂蚁集团蚂蚁天算安全实验室)