基于偏好的离线轨迹评估
Offline Preference-Based Trajectory Evaluation
AI总结：
针对离线评估中仅使用终端成功率导致统计效率低下的问题，提出基于偏好的轨迹评估方法，通过比较轨迹的时间偏好减少平局，提升区分能力、排名稳定性和数据效率。
摘要：
智能系统的离线评估通常将轨迹简化为终端成功，丢弃了部分进展信息并导致大量平局，通过减少有效样本量和削弱区分系统的能力，造成显著的统计低效。我们提出基于偏好的轨迹评估，该方法通过时间偏好（关于进展和返回时间分布）直接比较轨迹。我们发现，在多种智能和交互基准测试中，基于标准成功率的指标在大约75%的实例上产生平局比较，而轨迹感知偏好将平局减少到大约35%，从而提高了区分能力、排名稳定性和数据效率。我们的结果表明，通常归因于数据收集不足或问题难度的基准饱和，也可能由评估指标的选择所解释。
英文摘要：
Offline evaluation of agentic systems often collapses trajectories to terminal success, discarding information about partial progress and inducing widespread ties, creating substantial statistical inefficiency by reducing effective sample size and weakening the ability to distinguish systems. We propose preference-based trajectory evaluation, which compares trajectories directly through temporal preferences over progress and time-to-return profiles. We find that, across diverse agentic and interactive benchmarks, standard success-based metrics produce tied comparisons on roughly 75% of instances, whereas trajectory-aware preferences reduce ties to roughly 35%, improving discriminative power, ranking stability, and data efficiency. Our results suggest that benchmark saturation, often attributed to poor data collection or problem difficulty, may also be explained by the choice of evaluation measure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
