REPLUG：检索增强的黑盒语言模型
REPLUG: Retrieval-Augmented Black-Box Language Models
AI总结：
提出REPLUG检索增强框架，将LM视为黑盒并前置检索文档，同时利用LM监督检索模型，在GPT-3和Codex上显著提升了语言建模与MMLU性能。
摘要：
我们引入了REPLUG，一个检索增强的语言建模框架，它将语言模型（LM）视为黑盒，并通过可调的检索模型对其进行增强。与以往使用特殊交叉注意力机制训练语言模型来编码检索文本的检索增强LM不同，REPLUG只需将检索到的文档前置到冻结的黑盒LM的输入中。这种简单的设计可以轻松应用于任何现有的检索和语言模型。此外，我们展示了LM可用于监督检索模型，进而检索模型能找到帮助LM做出更好预测的文档。实验表明，带有微调检索器的REPLUG将GPT-3（175B）在语言建模上的性能显著提升了6.3%，并将Codex在五样本MMLU上的性能提升了5.1%。
英文摘要：
We introduce REPLUG, a retrieval-augmented language modeling framework that treats the language model (LM) as a black box and augments it with a tuneable retrieval model. Unlike prior retrieval-augmented LMs that train language models with special cross attention mechanisms to encode the retrieved text, REPLUG simply prepends retrieved documents to the input for the frozen black-box LM. This simple design can be easily applied to any existing retrieval and language models. Furthermore, we show that the LM can be used to supervise the retrieval model, which can then find documents that help the LM make better predictions. Our experiments demonstrate that REPLUG with the tuned retriever significantly improves the performance of GPT-3 (175B) on language modeling by 6.3%, as well as the performance of Codex on five-shot MMLU by 5.1%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
