arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.02598cs.DLcs.AI
收藏

LongDA:评估基于LLM的代理在长文档数据分析中的基准测试

LongDA: Benchmarking LLM Agents for Long-Document Data Analysis

University of Notre Dame(圣母大学);IBM Research(IBM研究院);University of Connecticut(康涅狄格大学)

Yiyang Li, Zheyuan Zhang, Tianyi Ma, Zehong Wang, Keerthiram Murugesan, Chuxu Zhang, Yanfang Ye

AI总结:

LongDA通过评估基于LLM的代理在长文档数据分析中的性能,揭示了现有模型在处理复杂任务时的不足。

摘要:

我们介绍了LongDA,一个用于评估基于LLM的代理在文档密集型分析工作流中的数据分析基准。与现有基准假设良好规范的模式和输入不同,LongDA针对现实世界中导航长文档和复杂数据为主要瓶颈的场景。为此,我们手动整理了来自17个公开可用的美国国家调查的原始数据文件、长而异质的文档和专家撰写的出版物,从中提取了505个基于真实分析实践的分析查询。解决这些查询需要代理首先从多个无结构文档中检索和整合关键信息,然后执行多步骤计算并编写可执行代码,这仍然对现有数据分析代理构成挑战。为了支持在此设置下的系统评估,我们开发了LongTA,一个工具增强的代理框架,使文档访问、检索和代码执行成为可能,并评估了一系列专有和开源模型。我们的实验揭示了即使在最先进模型之间也存在显著的性能差距,突显了研究人员在将LLM代理应用于现实世界、高风险分析设置中的决策支持之前应考虑的挑战。

英文摘要:

We introduce LongDA, a data analysis benchmark for evaluating LLM-based agents under documentation-intensive analytical workflows. In contrast to existing benchmarks that assume well-specified schemas and inputs, LongDA targets real-world settings in which navigating long documentation and complex data is the primary bottleneck. To this end, we manually curate raw data files, long and heterogeneous documentation, and expert-written publications from 17 publicly available U.S. national surveys, from which we extract 505 analytical queries grounded in real analytical practice. Solving these queries requires agents to first retrieve and integrate key information from multiple unstructured documents, before performing multi-step computations and writing executable code, which remains challenging for existing data analysis agents. To support the systematic evaluation under this setting, we develop LongTA, a tool-augmented agent framework that enables document access, retrieval, and code execution, and evaluate a range of proprietary and open-source models. Our experiments reveal substantial performance gaps even among state-of-the-art models, highlighting the challenges researchers should consider before applying LLM agents for decision support in real-world, high-stakes analytical settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑