基于表层检索降低检索增强语言模型的困惑度
Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Models
AI总结：
本文发现Retro模型的性能增益主要来自表层相似性，提出用BM25替换其语义检索，显著降低困惑度，并通过重排序以较小计算代价保持效果。
摘要：
为语言模型增加检索机制已被证明能在保持较少参数量的同时显著提升性能。检索增强模型通常依赖一种语义检索机制，该机制基于查询块与潜在邻居的稠密表示之间的相似度。本文研究了当前最先进的Retro模型，并观察到其性能增益更宜用表层相似性（如token重叠）来解释。受此启发，我们将Retro中的语义检索替换为基于BM25的表层方法，从而显著降低了困惑度。由于在大规模数据集上进行完整BM25检索的计算代价可能较高，我们还在重排序场景中应用该方法，以最小的计算开销获得了部分困惑度降低。
英文摘要：
Augmenting language models with a retrieval mechanism has been shown to significantly improve their performance while keeping the number of parameters low. Retrieval-augmented models commonly rely on a semantic retrieval mechanism based on the similarity between dense representations of the query chunk and potential neighbors. In this paper, we study the state-of-the-art Retro model and observe that its performance gain is better explained by surface-level similarities, such as token overlap. Inspired by this, we replace the semantic retrieval in Retro with a surface-level method based on BM25, obtaining a significant reduction in perplexity. As full BM25 retrieval can be computationally costly for large datasets, we also apply it in a re-ranking scenario, gaining part of the perplexity reduction with minimal computational overhead.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
