跨文本跨度审计基于代理的验证
Auditing Proxy-Based Validation Across Text Spans
浏览论文内容
中文总结 AI 辅助
针对代理验证中共享文本跨度导致虚假一致性的问题，提出验证契约与跨度外控制方法，实验证明短跨度下代理一致性不可靠，并呼吁声明跨度、报告构念一致性及发布生成内容。
中文摘要 AI 辅助
评估分数通常通过与廉价代理标签的一致性来验证。然而，当分数和代理从同一文本跨度计算时，这种一致性可能源于两者共享的表面证据，而非代理旨在代表的语义构念。我们通过将分数、其跨度、代理和目标构念声明为验证契约，然后在该评分跨度之外严格重新评估该代理规则，使这一区别明确化。在仅改变共享文本边界的受控HotpotQA正确性实验中，分数与其代理的一致性远高于与正确性在50字符前缀处的一致性：差距为+0.184，从120字符起缩小至至多+0.045。在该短前缀处，分数仍能预测答案字符串是否稍后出现（AUC 0.634），而等价检验将其与正确性的一致性置于随机水平，因此所报告的代理一致性并不能确立分数对正确性进行排序。在OR-Bench上，抑制每个模型重复出现的开场模板消除了分数与拒绝代理的大部分关联，而匹配体积删除几乎未消除任何关联，且构念一致性仍处于随机水平。十一个外部契约中仅三个支持跨度外控制，我们采样的路由研究均未发布其所需的生成内容。因此，我们要求基于代理的验证声明应声明每个标签读取的跨度，在代理一致性旁边报告构念一致性，并发布允许在评分跨度外重新读取代理的生成内容。
英文摘要
Evaluation scores are often validated by their agreement with inexpensive proxy labels. When the score and the proxy are computed from the same text span, however, that agreement can arise from surface evidence the two share rather than from the semantic construct the proxy is meant to represent. We make the distinction explicit by declaring the score, its span, the proxy and the target construct as a validation contract, then re-evaluating that proxy rule strictly outside the scored span. In a controlled HotpotQA correctness experiment varying only the shared text boundary, the score agrees with its proxy far better than with correctness at a 50-character prefix: the gap is +0.184, collapsing to at most +0.045 from 120 characters onward. At that short prefix the score still predicts whether the answer string appears later (AUC 0.634) while an equivalence test places its agreement with correctness at chance, so the reported proxy agreement does not establish that the score ranks correctness. On OR-Bench, suppressing each model's recurring opening templates removes most of the score's association with the refusal proxy, while matched-volume deletion removes almost none and construct agreement stays at chance. Only three of eleven external contracts support the off-span control, and none of the routing studies we sampled released the generations it needs. We therefore ask that a proxy-based validation claim declare the span each label is read from, report the construct agreement beside the proxy agreement, and release the generations that let the proxy be re-read off the scored span.
发表机构
- Kookmin University（国民大学）
- UStechlab
机构由 AI 辅助整理，请以论文原文为准。
