MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations
S-Lab, Nanyang Technological University(南洋理工大学S-Lab);Peking University(北京大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);National University of Singapore(新加坡国立大学);Wuhan University(武汉大学);Singapore Management University(新加坡管理大学);Shanghai AI Laboratory(上海人工智能实验室);University of Arizona(亚利桑那大学);Fudan University(复旦大学)
Understanding documents with rich layouts and multi-modal components is a long-standing and practical task. Recent Large Vision-Language Models (LVLMs) have made remarkable strides in various tasks, particularly in single-page document understanding (DU). However, their abilities on long-context DU remain an open problem. This work presents MMLongBench-Doc, a long-context, multi-modal benchmark comprising 1,062 expert-annotated questions. Distinct from previous datasets, it is constructed upon 130 lengthy PDF-formatted documents with an average of 49.4 pages and 20,971 textual tokens. Towards comprehensive evaluation, answers to these questions rely on pieces of evidence from (1) different sources (text, image, chart, table, and layout structure) and (2) various locations (i.e. page number). Moreover, 33.2% of the questions are cross-page questions requiring evidence across multiple pages. 22.8% of the questions are designed to be unanswerable for detecting potential hallucinations. Experiments on 14 LVLMs demonstrate that long-context DU greatly challenges current models. Notably, the best-performing model, GPT-4o, achieves an F1 score of only 42.7%, while the second-best, GPT-4V, scores 31.4%. Furthermore, 12 LVLMs (all except GPT-4o and GPT-4V) even present worse performance than their LLM counterparts which are fed with lossy-parsed OCR documents. These results validate the necessity of future research toward more capable long-context LVLMs. Project Page: https://mayubo2333.github.io/MMLongBench-Doc
MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly
CSE Department, HKUST(香港科技大学计算机科学与工程系);Tencent AI Seattle Lab(腾讯AI西雅图实验室);University of Edinburgh(爱丁堡大学);NVIDIA AI Technology Center (NVAITC), NVIDIA, Santa Clara, USA(英伟达圣克拉拉人工智能技术中心)
DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding
School of Computer Science, National Engineering Research Center for Multimedia Software and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China(计算机学院,国家多媒体软件工程技术研究中心和湖北多媒体与网络通信工程重点实验室,武汉大学,中国);Alibaba Group, Hangzhou, China(阿里巴巴集团,杭州,中国);Independent Researcher(独立研究者);Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence, United Arab Emirates(机器学习系,Mohamed bin Zayed人工智能大学,阿拉伯联合酋长国)
HighlightBench: Benchmarking Markup-Driven Table Reasoning in Scientific Documents
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of California, Merced(加州大学默塞德分校);University of Queensland(昆士兰大学)
DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments
Hong Kong Baptist University(香港浸会大学);University of Science and Technology of China(中国科学技术大学);The Hong Kong University of Science and Technology(香港科技大学)
MathDoc: Benchmarking Structured Extraction and Active Refusal on Noisy Mathematics Exam Papers
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);Gaotu Techedu Inc.(高途科技公司);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大模型与智能治理研究重点实验室);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(下一代智能搜索与推荐工程研究中心,教育部);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);University of Science and Technology of China(中国科学技术大学)
One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support
University of Bern(伯尔尼大学);University of Copenhagen(哥本哈根大学);Bern University of Applied Sciences(伯尔尼应用科学大学);University of Fribourg(弗里堡大学);Stanford University(斯坦福大学)
UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents
School of Computer Science and Engineering, Northeastern University, Shenyang, China(东北大学计算机科学与工程学院);Department of Computer Science and Technology, Tsinghua University, Beijing, China(清华大学计算机科学与技术系);Alibaba Group, Hangzhou, China(阿里巴巴集团)
TAP-RAG: Task-Aware Policy Control for Long-Document Multimodal Question Answering
School of Electrical and Information Engineering, Tianjin University(天津大学电气与信息工程学院);The International Joint Institute of Tianjin University(天津大学国际联合学院);Department of Automation, Tsinghua University(清华大学自动化系)
Q-Doc: Benchmarking Document Image Quality Assessment Capabilities in Multi-modal Large Language Models
South China Normal University(华南师范大学);Nanyang Technological University(南洋理工大学);City University of Hong Kong(香港城市大学);Shenzhen Academy of Inspection and Quarantine(深圳检验检疫局);Shenzhen University(深圳大学)
DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents
University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Massachusetts Institute of Technology(麻省理工学院);Stanford University(斯坦福大学);Kookmin University(国民大学);MIT-IBM Watson AI Lab, IBM Research(麻省理工学院-IBM沃森人工智能实验室,IBM研究院)
FCMBench-Video: Benchmarking Document Video Intelligence
AI Lab, Qifu Technology(启赋科技AI实验室);College of Future Information Technology, Fudan University(复旦大学未来信息学院);School of Future Technology, South China University of Technology(华南理工大学未来技术学院);Pazhou Lab(琶洲实验室)
Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Framework
College of Computer Science, Chongqing University(重庆大学计算机学院);Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);Kuaishou Technology(快手科技)
Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs
Institute for Machine Learning and Analytics (IMLA), Offenburg University, Offenburg, Germany(机器学习与分析研究所(IMLA)、奥芬堡大学);University of Mannheim, Mannheim, Germany(曼海姆大学)
MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents
INSAIT;Lanzhou University(兰州大学);King Abdullah University of Science and Technology(国王阿卜杜勒-阿齐兹科学与技术大学);Heriot-Watt University(赫瑞-沃德大学);Monash University(莫纳什大学);University College Dublin(都柏林大学学院)