在大规模文档集合中导航:MuDABench用于多文档分析问答
Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA
AI总结:
本文提出在大规模半结构化文档集合上进行分析问答的任务,介绍了MuDABench多文档分析问答基准,要求跨多个文档提取和综合信息以进行定量分析,实验发现标准RAG系统表现不佳,提出多代理工作流以提升性能。
摘要:
本文介绍了在大规模、半结构化文档集合上进行分析问答的任务。我们提出了MuDABench,一个用于多文档分析问答的基准,其中问题需要跨多个文档提取和综合信息以执行定量分析。与现有多文档问答基准通常仅需少量文档信息且跨文档推理有限不同,MuDABench要求广泛的跨文档分析和汇总。通过远距离监督利用文档级元数据和标注的金融数据库构建,MuDABench包含超过80,000页和332个分析问答实例。我们还提出了一种评估协议,测量最终答案的准确性,并利用中间事实覆盖作为推理过程的辅助诊断信号。实验表明,标准RAG系统将所有文档视为扁平检索池表现不佳。为解决这些限制,我们提出了一种多代理工作流,协调规划、提取和代码生成模块。尽管这种方法显著提高了过程和结果指标,但与人类专家表现相比仍有显著差距。我们的分析识别出两个主要瓶颈:单文档信息提取准确性和当前系统中不足的领域特定知识。MuDABench可在https://github.com/Zhanli-Li/MuDABench上获取。
英文摘要:
This paper introduces the task of analytical question answering over large, semi-structured document collections. We present MuDABench, a benchmark for multi-document analytical QA, where questions require extracting and synthesizing information across numerous documents to perform quantitative analysis. Unlike existing multi-document QA benchmarks that typically require information from only a few documents with limited cross-document reasoning, MuDABench demands extensive inter-document analysis and aggregation. Constructed via distant supervision by leveraging document-level metadata and annotated financial databases, MuDABench comprises over 80,000 pages and 332 analytical QA instances. We also propose an evaluation protocol that measures final answer accuracy and uses intermediate-fact coverage as an auxiliary diagnostic signal for the reasoning process. Experiments reveal that standard RAG systems, which treat all documents as a flat retrieval pool, perform poorly. To address these limitations, we propose a multi-agent workflow that orchestrates planning, extraction, and code generation modules. While this approach substantially improves both process and outcome metrics, a significant gap remains compared to human expert performance. Our analysis identifies two primary bottlenecks: single-document information extraction accuracy and insufficient domain-specific knowledge in current systems. MuDABench is available at https://github.com/Zhanli-Li/MuDABench.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。