We introduce LongDA, a data analysis benchmark for evaluating LLM-based agents under documentation-intensive analytical workflows. In contrast to existing benchmarks that assume well-specified schemas and inputs, LongDA targets real-world settings in which navigating long documentation and complex data is the primary bottleneck. To this end, we manually curate raw data files, long and heterogeneous documentation, and expert-written publications from 17 publicly available U.S. national surveys, from which we extract 505 analytical queries grounded in real analytical practice. Solving these queries requires agents to first retrieve and integrate key information from multiple unstructured documents, before performing multi-step computations and writing executable code, which remains challenging for existing data analysis agents. To support the systematic evaluation under this setting, we develop LongTA, a tool-augmented agent framework that enables document access, retrieval, and code execution, and evaluate a range of proprietary and open-source models. Our experiments reveal substantial performance gaps even among state-of-the-art models, highlighting the challenges researchers should consider before applying LLM agents for decision support in real-world, high-stakes analytical settings.
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所 MAIS);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Alibaba Group(阿里巴巴集团)
DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents
University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Massachusetts Institute of Technology(麻省理工学院);Stanford University(斯坦福大学);Kookmin University(国民大学);MIT-IBM Watson AI Lab, IBM Research(麻省理工学院-IBM沃森人工智能实验室,IBM研究院)
Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Framework
College of Computer Science, Chongqing University(重庆大学计算机学院);Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);Kuaishou Technology(快手科技)
Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs
Institute for Machine Learning and Analytics (IMLA), Offenburg University, Offenburg, Germany(机器学习与分析研究所(IMLA)、奥芬堡大学);University of Mannheim, Mannheim, Germany(曼海姆大学)
SkillScriptBench: Benchmarking Self-Evolution of Executable Agent Skill Packages Beyond Markdown
The Hong Kong University of Science and Technology(香港科技大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Harbin Institute of Technology, Harbin(哈尔滨工业大学)
One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support
University of Bern(伯尔尼大学);University of Copenhagen(哥本哈根大学);Bern University of Applied Sciences(伯尔尼应用科学大学);University of Fribourg(弗里堡大学);Stanford University(斯坦福大学)
ClinOCR-Bench: A Comprehensive Clinical Scanned Document Dataset for Optical Character Recognition Model Evaluation
McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston(德克萨斯大学健康科学中心休斯顿分校麦威廉斯生物医学信息学学院);Enterprise Development & Integration, University of Texas MD Anderson Cancer Center(德克萨斯大学MD安德森癌症中心企业开发与集成)