arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.21230cs.CL
收藏

TRACE: 基于轨迹的深度研究代理综合评估

TRACE: Trajectory-Aware Comprehensive Evaluation for Deep Research Agents

The Chinese University of Hong Kong(香港中文大学)

Yanyu Chen, Jiyue Jiang, Jiahong Liu, Yifei Zhang, Xiao Guo, Irwin King

AI总结:

TRACE提出基于轨迹的深度研究代理综合评估框架,通过分层效用函数和阶梯式能力评估协议,解决单一指标评估的不足,揭示代理在准确性、效率和鲁棒性之间的关键权衡。

摘要:

深度研究代理的评估是一个关键挑战,因为传统基于结果的指标无法捕捉其复杂推理的细微差别。当前评估面临两大主要挑战:1)依赖单一指标如Pass@1,造成"高分 illusion",忽略了推理过程的质量、效率和合理性;2)静态基准无法量化诸如鲁棒性和潜在能力等关键属性。为解决这些差距,我们引入TRACE(基于轨迹的综合评估)框架,全面评估整个问题解决轨迹。为对抗"高分 illusion",我们提出分层轨迹效用函数,量化过程效率和认知质量,包括证据基础,同时包括准确性。为衡量更深层次属性,TRACE引入了阶梯式能力评估协议,通过确定成功所需的最小指导来量化代理的潜在能力。我们的贡献包括TRACE框架、其新颖的指标以及配套的DeepResearch-Bench可调节复杂性。实验表明,TRACE提供了细粒度的排名,揭示了代理准确性、效率和鲁棒性之间完全被单一指标遗漏的关键权衡。

英文摘要:

The evaluation of Deep Research Agents is a critical challenge, as conventional outcome-based metrics fail to capture the nuances of their complex reasoning. Current evaluation faces two primary challenges: 1) a reliance on singular metrics like Pass@1, creating a "high-score illusion" that ignores the quality, efficiency, and soundness of the reasoning process; and 2) the failure of static benchmarks to quantify crucial attributes like robustness and latent capability. To address these gaps, we introduce TRACE (Trajectory-Aware Comprehensive Evaluation), a framework that holistically assesses the entire problem-solving trajectory. To counter the "high-score illusion", we propose a Hierarchical Trajectory Utility Function that quantifies process efficiency and cognitive quality, including evidence grounding, alongside accuracy. To measure deeper attributes, TRACE introduces a Scaffolded Capability Assessment protocol, quantifying an agent's latent ability by determining the minimum guidance needed for success. Our contributions include the TRACE framework, its novel metrics, and the accompanying DeepResearch-Bench with controllable complexity. Experiments show TRACE delivers a granular ranking that uncovers critical trade-offs between agent accuracy, efficiency, and robustness entirely missed by singular metrics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑