arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.06758cs.CL
收藏

长上下文与检索增强语言模型中证据利用的四条件诊断协议

Diagnosing Evidence Utilization in Long-Context and Retrieval-Augmented Language Models under Matched Evidence Conditions

University of Western Ontario(西方大学)

Haizhou Xia

AI总结:

提出四条件证据可用性协议,通过ONCU估计器分离无证据、全上下文、检索证据和Oracle证据四种条件下的模型表现,诊断长上下文与检索增强语言模型的证据利用瓶颈。

摘要:

最终答案准确性、检索召回率和引用重叠本身并不能确定长上下文或检索增强语言模型是否使用了所提供的证据。模型可能从参数记忆中进行回答,尽管接收到正确的段落却失败,或者引用证据但未将其转换为所请求的答案。本文提出了一种匹配的四条件证据可用性协议——无证据、全上下文、检索证据和Oracle证据参考——用于在固定示例、提示、评分字段、检索设置和有效性检查下诊断证据利用情况。ONCU被用作协议绑定的估计器,用于估计恢复的Oracle参考证据优势,并且仅针对分母有效的组进行计算;无分母的答案、证据、检索和失败审计指标分别报告。实证研究评估了来自Qwen、Gemma、Llama和Mistral家族的五个本地开源模型,在Controlled-ONCU-safe16K、HotpotQA-ONCU和2WikiMultiHopQA-ONCU上进行了评估,共产生18,000个ONCU兼容预测。主要发现是任务相关的瓶颈分裂:受控合成设置主要暴露全上下文利用失败,而测试的真实多跳设置主要暴露无分母答案和证据指标中的检索链覆盖失败,ONCU在Oracle改进组上支持相同方向。贡献在于提供了一个诊断协议,用于分离无证据可回答性、Oracle证据可恢复性、全上下文利用和检索条件利用,而不是为长上下文或检索增强系统提供单一分数排行榜。

英文摘要:

Final-answer accuracy, retrieval recall, and citation overlap do not reveal how much answer advantage a long-context or retrieval-augmented language model actually recovers from supplied evidence. A model may answer from parametric priors, fail to use evidence that is present, or cite relevant text without converting it into the final answer. This paper introduces a four-condition diagnostic protocol for evidence-utilization evaluation under matched examples, models, prompts, and scoring rules. The protocol compares no-evidence, full-context, retrieved-evidence, and oracle-evidence reference conditions, and uses Oracle-Reference Normalized Context Utilization (ONCU) as a denominator-valid estimate of recovered oracle-reference evidence advantage. The empirical study evaluates five local open-weight models from the Qwen, Gemma, Llama, and Mistral families over Controlled-ONCU-safe16K, HotpotQA-ONCU, and 2WikiMultiHopQA-ONCU, comprising 18,000 ONCU-compatible predictions. Results show a task-dependent diagnostic pattern: controlled synthetic settings expose reduced recovery when the same evidence is embedded in long input rather than supplied compactly, while realistic multi-hop reconstructions show that full-context inputs outperform the tested retrieved inputs in denominator-free answer and evidence metrics, with ONCU supporting the same direction on oracle-improving groups. Sensitivity audits with stronger retrieval settings narrow some gaps but do not overturn the scoped interpretation. The main contribution is therefore not a single utilization ratio, but a matched diagnostic protocol that separates no-evidence answerability, oracle-evidence recoverability, full-context recovery, retrieval-conditioned recovery, denominator validity, and companion answer/evidence diagnostics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑