AD-Bench: 面向LLM智能体的真实世界、轨迹感知广告分析基准
AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents
AI总结：
提出AD-Bench基准，通过动态真实答案管道和轨迹感知评估，衡量LLM智能体在真实广告分析任务中的多轮工具协作能力，发现最优模型在复杂任务上仍有显著差距。
摘要：
尽管大语言模型（LLM）智能体在复杂推理方面取得了显著进展，但在真实环境中评估它们仍然是一个开放问题。现有基准大多局限于理想化模拟，未能涵盖广告和营销分析等专业领域，这些领域的任务需要与专业工具进行多轮交互，并且随着数据和平台规则的演变，真实答案很快就会过时。为了解决这个问题，我们提出了AD-Bench，这是一个基于生产广告平台上真实用户营销分析请求构建的基准。AD-Bench引入了两个关键设计：（i）一个动态真实答案管道，它重放专家工具调用轨迹以生成与当前环境一致的答案，缓解答案过时问题；（ii）一个轨迹感知评估，联合衡量端到端答案正确性（Pass@k）和轨迹覆盖率。请求被分为三个难度级别（L1-L3），以探索多轮、多工具协作。实验表明，最佳模型Claude-Opus-4.7在整体上达到了Pass@1 = 76.9%、Pass@3 = 80.4%和82.7%的轨迹覆盖率，但在L3上急剧下降至Pass@1 = 61.4%和Pass@3 = 65.1%，表明即使是最先进的智能体在复杂广告分析中仍存在显著差距。
英文摘要：
While Large Language Model (LLM) agents have made remarkable progress on complex reasoning, evaluating them in real-world environments remains an open problem. Existing benchmarks are largely confined to idealized simulations and fail to capture specialized domains such as advertising and marketing analytics, where tasks require multi-round interaction with professional tools and where ground-truth answers quickly become obsolete as data and platform rules evolve. To address this, we propose AD-Bench, a benchmark built from real user marketing-analysis requests on a production advertising platform. AD-Bench introduces two key designs: (i) a dynamic ground-truth pipeline that replays expert tool-call trajectories to regenerate answers consistent with the current environment, mitigating answer obsolescence; and (ii) a trajectory-aware evaluation that jointly measures end-to-end answer correctness (Pass@k) and trajectory coverage. Requests are stratified into three difficulty levels (L1-L3) to probe multi-round, multi-tool collaboration. Experiments show that the best model, Claude-Opus-4.7, attains Pass@1 = 76.9% and Pass@3 = 80.4% with 82.7% trajectory coverage overall, yet drops sharply on L3 to Pass@1 = 61.4% and Pass@3 = 65.1%, revealing that even state-of-the-art agents have substantial gaps in complex advertising analytics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
