arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.04482cs.HC
收藏

ProAgentBench:基于真实数据评估LLM代理的前瞻性帮助

ProAgentBench: Evaluating LLM Agents for Proactive Assistance with Real-World Data

Tsinghua Shenzhen Graduate School, Tsinghua University(清华大学深圳研究生院)

Yuanbo Tang, Huaze Tang, Tingyu Cao, Lam Nguyen, Anping Zhang, Xinwen Cao, Chunkang Liu, Wenbo Ding, Yang Li

AI总结:

ProAgentBench通过真实用户数据构建基准,评估LLM代理在连续工作流程中的前瞻性帮助能力,揭示真实数据对预测准确性的提升作用。

摘要:

前瞻性代理能够在没有明确提示的情况下预测用户意图,代表了人机交互的重要进展,有望减少认知负荷并优化工作流程。然而,现有数据集存在两个关键缺陷:(1)依赖LLM合成数据,无法捕捉真实人类决策模式;(2)专注于孤立任务而非连续工作流程,缺失学习前瞻性干预信号所需的前置帮助行为上下文。为解决这些不足,我们引入ProAgentBench,一个用于工作场景中前瞻性代理的严格基准。我们的贡献包括:(1)一个分层任务框架,将前瞻性帮助分解为时间预测和帮助内容生成;(2)一个符合隐私的数据集,包含来自500多个小时真实用户会话的28000多个事件,保留了合成数据中缺失的突发交互模式(burstiness B=0.787);(3)广泛的实验评估LLM和VLM基线。从数值上看,长期记忆和历史上下文显著提高了预测准确性,而真实世界训练数据在很大程度上优于合成替代品。我们通过https://anonymous.4open.science/r/ProAgentBench-6BC0发布我们的数据集和代码。

英文摘要:

Proactive agents that anticipate user intentions without explicit prompts represent a significant evolution in human-AI interaction, promising to reduce cognitive load and streamline workflows. However, existing datasets suffer from two critical deficiencies: (1) reliance on LLM-synthesized data that fails to capture authentic human decision-making patterns, and (2) focus on isolated tasks rather than continuous workflows, missing the pre-assistance behavioral context essential for learning proactive intervention signals. To address these gaps, we introduce ProAgentBench, a rigorous benchmark for proactive agents in working scenarios. Our contributions include: (1) a hierarchical task framework that decomposes proactive assistance into timing prediction and assist content generation; (2) a privacy-compliant dataset with 28,000+ events from 500+ hours of real user sessions, preserving bursty interaction patterns (burstiness B=0.787) absent in synthetic data; and (3) extensive experiments that evaluates LLM- and VLM-based baselines. Numerically, we showed that long-term memory and historical context significantly enhance prediction accuracy, while real-world training data substantially outperforms synthetic alternatives. We release our dataset and code at https://anonymous.4open.science/r/ProAgentBench-6BC0.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑