RAGScope：一种泄漏受控、成本感知的证据门控协议，用于RAG幻觉分诊
RAGScope: A Leakage-Controlled, Cost-Aware Evidence-Gating Protocol for RAG Hallucination Triage
浏览论文内容
中文总结 AI 辅助
提出RAGScope协议，通过泄漏受控的证据门评估，实现低成本RAG幻觉分诊，增强门RAGScope-E在精度和速度上优于基线，但需目标域校准。
中文摘要 AI 辅助
检索增强生成（RAG）系统需要低成本的方法来路由生成的答案：接受低风险的输出，审查不确定的输出，并将强大的验证器保留给昂贵的尾部。我们提出了RAGScope，一种泄漏受控的协议，用于评估仅使用任务输入、检索到的上下文和答案文本的局部证据门。该协议结合了上下文分组分割、折叠范围预处理、组自助法区间、部署操作点、端到端运行时间和显式源偏移压力测试。在三个RAGTruth任务上，增强门RAGScope-E在合并分组交叉验证中达到0.798的AUROC和0.660的平均精度（AP）。其合并AP超过ROUGE-L 0.034，95%上下文组区间为[0.002, 0.064]，尽管AUROC增益不显著，且ROUGE-L在数据到文本任务上仍然更强。在10%的审查预算下，RAGScope-E达到0.748的精确率；接受最低风险的50%产生0.141的残余不忠实度。RAGScope-E在CPU上运行时间为6.22毫秒/示例，而测试的DeBERTa-NLI和HHEM设置分别为145.75和223.07毫秒/示例。一个14,900示例的HaluBench压力测试暴露了部署边界：域内校准的门达到0.879的AUROC，但留源校准平均仅为0.466。仅目标校准在每源100个标签时恢复到0.675的AUROC，200个标签时恢复到0.685。因此，廉价证据门是有用的路由组件，但学习到的校准必须在目标域内进行验证和调整。
英文摘要
Retrieval-augmented generation (RAG) systems need inexpensive ways to route generated answers: accept low-risk outputs, review uncertain ones, and reserve strong verifiers for the expensive tail. We present RAGScope, a leakage-controlled protocol for evaluating local evidence gates that use only the task input, retrieved context, and answer text. The protocol combines context-grouped splits, fold-scoped preprocessing, group bootstrap intervals, deployment operating points, end-to-end runtime, and explicit source-shift stress tests. On three RAGTruth tasks, the enhanced gate RAGScope-E reaches 0.798 AUROC and 0.660 average precision (AP) in pooled grouped cross-validation. Its pooled AP exceeds ROUGE-L by 0.034 with a 95% context-group interval of [0.002, 0.064], although the AUROC gain is not significant and ROUGE-L remains stronger on data-to-text. At a top-10% review budget, RAGScope-E attains 0.748 precision; accepting the lowest-risk 50% yields 0.141 residual unfaithfulness. RAGScope-E runs in 6.22 ms/example on CPU, versus 145.75 and 223.07 ms/example for the tested DeBERTa-NLI and HHEM settings. A 14,900-example HaluBench stress test exposes the deployment boundary: an in-domain calibrated gate reaches 0.879 AUROC, but leave-source-out calibration averages only 0.466. Target-only calibration recovers to 0.675 AUROC with 100 labels per source and 0.685 with 200. Cheap evidence gates are therefore useful routing components, but learned calibration must be validated and adapted within the target domain.
发表机构
- Brown University（布朗大学）
- Georgia Institute of Technology（佐治亚理工学院）
机构由 AI 辅助整理，请以论文原文为准。
