CUB：语言模型上下文利用技术的基准测试
CUB: Benchmarking Context Utilisation Techniques for Language Models
AI总结：
本文提出CUB基准，用于评估语言模型在不同噪声上下文下的上下文利用技术，发现现有方法在真实场景中存在不足，需更全面的测试。
摘要：
在知识密集型任务中，整合外部知识至关重要，但语言模型可能忽略与过时参数记忆矛盾的相关信息或被无关上下文干扰。尽管许多上下文利用技术（CMTs）已被提出以缓解这些问题，但缺乏系统比较。本文开发了CUB（上下文利用基准）——首个全面的基准，用于在检索增强生成（RAG）中诊断CMTs在多样化噪声上下文条件下的表现。通过该基准，我们对七种最先进的方法进行了迄今为止最全面的评估，这些方法代表了CMTs的主要类别，应用于三个多样化数据集和任务，以及11个语言模型。我们的发现揭示了当前CMT评估实践中的关键差距，证明了需要整体测试的必要性。我们发现，大多数现有CMTs难以处理真实世界RAG场景中遇到的完整上下文类型。我们还发现，许多CMTs在简单合成数据集上表现良好，但相比之下，在包含自然样本的真实数据集上表现较差。
英文摘要：
Incorporating external knowledge is crucial for knowledge-intensive tasks, such as question answering and fact checking. However, language models (LMs) may ignore relevant information that contradicts outdated parametric memory or be distracted by irrelevant contexts. While many context utilisation manipulation techniques (CMTs) have recently been proposed to alleviate these issues, few have seen systematic comparison. In this paper, we develop CUB (Context Utilisation Benchmark) - the first comprehensive benchmark designed to help diagnose CMTs under diverse noisy context conditions within retrieval-augmented generation (RAG). With this benchmark, we conduct the most extensive evaluation to date of seven state-of-the-art methods, representative of the main categories of CMTs, across three diverse datasets and tasks, applied to 11 LMs. Our findings expose critical gaps in current CMT evaluation practices, demonstrating the need for holistic testing. We reveal that most existing CMTs struggle to handle the full spectrum of context types encountered in real-world RAG scenarios. We also find that many CMTs display inflated performance on simple synthesised datasets, compared to more realistic datasets with naturally occurring samples.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
