arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.11513cs.CL
收藏

小型语言模型能利用它们检索到的信息吗?不同模型规模下的检索利用经验研究

Can Small Language Models Use What They Retrieve? An Empirical Study of Retrieval Utilization Across Model Scale

Sanchit Pandey

AI总结:

研究发现小型语言模型在检索信息时存在利用瓶颈,无法有效提取答案,且上下文干扰导致错误生成,表明RAG在小模型中效果有限。

摘要:

检索增强生成RAG被广泛用于提高语言模型的事实准确性,但仍然不清楚较小的模型(7B参数或更少)是否能有效利用检索到的信息。为研究这个问题,我们评估了从360M到8B的五个模型规模,分别属于SmolLM2、Qwen2.5和Llama 3.1三种架构家族,在四种检索条件下进行测试,包括无检索、BM25检索、使用E5 large v2的密集检索以及oracle检索(检索的段落保证包含答案)。我们引入了一个参数化的知识拆分,将模型本身能够回答的问题与需要外部知识的问题分开,这使我们能够将利用失败与检索质量失败区分开来。我们发现三个主要结果。首先,即使在oracle检索条件下,7B参数或更小的模型在无法单独回答的问题上也只有85到100%的时间无法提取正确答案,这表明存在根本性的利用瓶颈。其次,添加检索上下文会破坏42到100%的答案,这些答案模型之前已经知道,这表明是由上下文的存在而非其质量所驱动的干扰效应。第三,对2588个oracle失败案例的错误分析显示,主导的失败模式是无关生成,即模型完全忽略了提供的上下文。这些模式在多种提示模板和检索方法下都成立。结果表明,对于7B参数以下的模型,RAG的主要限制是上下文利用而非检索质量,并且在标准评估条件下,部署RAG在这一规模可能会导致净负面效益。

英文摘要:

Retrieval augmented generation RAG is widely deployed to improve factual accuracy in language models yet it remains unclear whether smaller models of size 7B parameters or less can effectively utilize retrieved information. To investigate this question we evaluate five model sizes from 360M to 8B across three architecture families SmolLM2 Qwen2.5 and Llama 3.1 under four retrieval conditions including no retrieval BM25 dense retrieval using E5 large v2 and oracle retrieval where the retrieved passage is guaranteed to contain the answer. We introduce a parametric knowledge split that separates questions a model can already answer from those that require external knowledge which allows us to isolate utilization failure from retrieval quality failure. We find three main results. First even with oracle retrieval models of size 7B or smaller fail to extract the correct answer 85 to 100 percent of the time on questions they cannot answer alone which indicates a fundamental utilization bottleneck. Second adding retrieval context destroys 42 to 100 percent of answers the model previously knew suggesting a distraction effect driven by the presence of context rather than its quality. Third an error analysis of 2588 oracle failures shows that the dominant failure mode is irrelevant generation where the model ignores the provided context entirely. These patterns hold across multiple prompt templates and retrieval methods. The results indicate that for models below 7B parameters the main limitation of RAG is context utilization rather than retrieval quality and that deploying RAG at this scale can lead to a net negative trade off under standard evaluation conditions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑