智能体不确定性量化的适当评分规则
Proper Scoring Rules for Agentic Uncertainty Quantification
AI总结：
针对语言模型智能体轨迹中的不确定性信号，提出严格适当的轨迹评分规则TPS，用于评估逐步骤成功概率过程，并处理删失数据。
摘要：
语言模型智能体在轨迹中越来越多地发出不确定性信号，但现有的智能体不确定性量化评估常常混淆排序有用性与概率真实性。AUROC、AUPRC、风险覆盖、轨迹ECE和标量化轨迹评分评估了区分度、分箱校准或压缩摘要，但并未严格引出完整的基于前缀的条件成功概率轨迹$q_t = P^π(Y=1 | H_t)$。基于序列适当评分，我们引入了轨迹适当评分（TPS），这是一个预测器无关的严格适当的轨迹级评分规则族，适用于任何校准为最终成功概率的逐步骤不确定性信号。我们证明，在完全观测下，TPS在所选的评分族和权重方案内严格引出了成功概率过程。我们将构造扩展到行政删失轨迹，通过将完整数据评分投影到可观测的停止前缀上，得到精确的$q_Z$加权简化评分，并在$q_Z$未估计时得到可处理的近似。我们进一步表明，常见的轨迹评估器针对的是比完整前缀条件概率过程更弱的目标：轨迹ECE是分辨率盲的，而标量化轨迹Brier仅引出压缩标量，而非完整轨迹。在StrategyQA、Tau2-Bench、HotpotQA和WebShop上的实验表明，这些理论差异在操作上是可见的：概率重新校准可以显著改变TPS，而几乎不改变排序指标，并且可处理的删失近似相对于仅完整评估可能改变结论。
英文摘要：
Language-model agents increasingly emit uncertainty signals throughout a trajectory, but existing agentic UQ evaluations often conflate ranking usefulness with probabilistic truthfulness. AUROC, AUPRC, risk-coverage, Trajectory ECE, and scalarized trajectory scores evaluate discrimination, binwise calibration, or collapsed summaries, but do not strictly elicit the full prefix-conditioned success-probability trace $q_t = P^π(Y=1 | H_t)$. Building on prequential proper scoring, we introduce the Trajectory Proper Score (TPS), a predictor-agnostic family of strictly proper trajectory-level scoring rules for any per-step uncertainty signal calibrated into a probability of eventual success. We prove that TPS strictly elicits the success-probability process under complete observation, within the chosen score family and weight schedule. We extend the construction to administratively censored trajectories by projecting the complete-data score onto the observable stopped prefix, yielding an exact $q_Z$-weighted reduced score and a tractable approximation when $q_Z$ is unestimated. We further show that common trajectory evaluators target weaker objects than the full prefix-conditioned probability process: Trajectory ECE is resolution-blind, while scalarized Trajectory Brier elicits only the collapsed scalar, not the full trace. Experiments on StrategyQA, Tau2-Bench, HotpotQA, and WebShop show that these theoretical distinctions are operationally visible: probability recalibration can substantially change TPS while leaving rank metrics nearly unchanged, and the tractable censored approximation can change the verdict relative to complete-only evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
