理解文档是许多现实世界任务的核心，但仍是一个具有挑战性的课题。遗憾的是，目前尚未就如何全面评估文档理解能力达成共识，这严重阻碍了公平比较和衡量该领域的进展。为了对文档理解研究进行基准测试，本文总结了四项代表性能力，即文档分类、文档结构分析、文档信息提取和文档转录。在新的评估框架下，我们提出了文档语言理解评估（Document Language Understanding Evaluation，简称DLUE），这是一个涵盖各种形式、领域和文档体裁中广泛任务的新任务套件。我们还在DLUE上系统评估了六个成熟的transformer模型，发现由于内容冗长、底层结构复杂且知识分散，文档理解问题仍远未解决，目前尚无能在所有任务中占据主导地位的神经架构，这提出了对通用文档理解架构的需求。
英文摘要：
Understanding documents is central to many real-world tasks but remains a challenging topic. Unfortunately, there is no well-established consensus on how to comprehensively evaluate document understanding abilities, which significantly hinders the fair comparison and measuring the progress of the field. To benchmark document understanding researches, this paper summarizes four representative abilities, i.e., document classification, document structural analysis, document information extraction, and document transcription. Under the new evaluation framework, we propose \textbf{Document Language Understanding Evaluation} -- \textbf{DLUE}, a new task suite which covers a wide-range of tasks in various forms, domains and document genres. We also systematically evaluate six well-established transformer models on DLUE, and find that due to the lengthy content, complicated underlying structure and dispersed knowledge, document understanding is still far from being solved, and currently there is no neural architecture that dominates all tasks, raising requirements for a universal document understanding architecture.
NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens
Westlake University（西湖大学）；University of Waterloo（多伦多大学）；Hangzhou Normal University（杭州师范大学）；Shanghai AI Lab（上海人工智能实验室）；SJTU（上海交通大学）；NYU Shanghai（纽约大学上海分校）
UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents
School of Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程学院）；Department of Computer Science and Technology, Tsinghua University, Beijing, China（清华大学计算机科学与技术系）；Alibaba Group, Hangzhou, China（阿里巴巴集团）
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所 MAIS）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Alibaba Group（阿里巴巴集团）
MEBench: Benchmarking Large Language Models for Cross-Document Multi-Entity Question Answering
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；The Hong Kong University of Science and Technology（香港科学与技术大学）；China Mobile Information Technology Company Limited（中国移动信息技术有限公司）
MiLQ: Benchmarking IR Models for Bilingual Web Search with Mixed Language Queries
Graduate School of Artificial Intelligence, POSTECH, Republic of Korea（POSTECH人工智能研究生院）；Department of Computer Science and Engineering, POSTECH, Republic of Korea（POSTECH计算机科学与工程系）；Lilt, Inc.（Lilt公司）
本文提出首个真实且较受偏好的混合语言查询公开基准 MiLQ，评估多语言 IR 模型并揭示其表现不一致，同时验证英语混用可通过增强 token 匹配改善英文文档检索。
Redefining Simplicity: Benchmarking Large Language Models from Lexical to Document Simplification
School of Information and Engineering, Yangzhou University（扬州大学信息工程学院）；School of Computer Science and Information Technology, Hefei University of Technology（合肥工业大学计算机与信息学院）
Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction
University of Chinese Academy of Sciences（中国科学院大学）；Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）
Benchmarking LLMs on the Semantic Overlap Summarization Task
Research Center Trustworth Data Science and Security, Technical University of Dortmund（多特蒙德技术大学信任数据科学与安全研究中心）；School of Computing, Wichita State University（威斯康星州立大学计算机学院）；Department of Computer Science, NC State University（北卡罗来纳州立大学计算机科学系）
DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding
School of Computer Science, National Engineering Research Center for Multimedia Software and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China（计算机学院，国家多媒体软件工程技术研究中心和湖北多媒体与网络通信工程重点实验室，武汉大学，中国）；Alibaba Group, Hangzhou, China（阿里巴巴集团，杭州，中国）；Independent Researcher（独立研究者）；Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence, United Arab Emirates（机器学习系，Mohamed bin Zayed人工智能大学，阿拉伯联合酋长国）
Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies
Department of Computer Science and Engineering（计算机科学与工程系）；New York University（纽约大学）；New York, NY 10012（纽约市NY 10012）
MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations
S-Lab, Nanyang Technological University（南洋理工大学S-Lab）；Peking University（北京大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；National University of Singapore（新加坡国立大学）；Wuhan University（武汉大学）；Singapore Management University（新加坡管理大学）；Shanghai AI Laboratory（上海人工智能实验室）；University of Arizona（亚利桑那大学）；Fudan University（复旦大学）
DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments
Hong Kong Baptist University（香港浸会大学）；University of Science and Technology of China（中国科学技术大学）；The Hong Kong University of Science and Technology（香港科技大学）
