DeepSynth-Eval: 从客观角度评估深度调研写作中的信息整合
DeepSynth-Eval: Objectively Evaluating Information Consolidation in Deep Survey Writing
AI总结:
DeepSynth-Eval通过客观评估信息整合能力,揭示了深度调研写作中整合碎片信息的挑战,并证明代理计划与写作流程在提升报告质量方面优于单一生成方法。
摘要:
大型语言模型(LLMs)向自主代理演进推动了深度研究的进步。尽管检索能力已得到充分评估,但检索后的综合阶段——代理必须消化大量上下文并整合碎片化证据形成连贯的长篇报告——由于开放式写作的主观性而被低估。为弥合这一差距,我们引入DeepSynth-Eval,一个旨在客观评估信息整合能力的基准。我们利用高质量的综述论文作为黄金标准,通过逆向工程研究请求并从其参考文献中构建
英文摘要:
The evolution of Large Language Models (LLMs) towards autonomous agents has catalyzed progress in Deep Research. While retrieval capabilities are well-benchmarked, the post-retrieval synthesis stage--where agents must digest massive amounts of context and consolidate fragmented evidence into coherent, long-form reports--remains under-evaluated due to the subjectivity of open-ended writing. To bridge this gap, we introduce DeepSynth-Eval, a benchmark designed to objectively evaluate information consolidation capabilities. We leverage high-quality survey papers as gold standards, reverse-engineering research requests and constructing "Oracle Contexts" from their bibliographies to isolate synthesis from retrieval noise. We propose a fine-grained evaluation protocol using General Checklists (for factual coverage) and Constraint Checklists (for structural organization), transforming subjective judgment into verifiable metrics. Experiments across 96 tasks reveal that synthesizing information from hundreds of references remains a significant challenge. Our results demonstrate that agentic plan-and-write workflows significantly outperform single-turn generation, effectively reducing hallucinations and improving adherence to complex structural constraints.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。