RAVEN：结合检索增强的编码器-解码器语言模型的上下文学习
RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models
AI总结：
针对检索增强编码器-解码器语言模型上下文学习的预训练-推理不匹配与上下文长度受限问题，提出结合检索增强掩码语言建模和前缀语言建模的RAVEN模型及融合式上下文学习机制，在少样本场景下以更少参数取得媲美大模型的性能。
摘要：
本文研究了检索增强型编码器-解码器语言模型的上下文学习能力。我们首先对现有模型进行了全面分析，发现它们在上下文学习方面存在局限性，主要源于预训练与推理之间的不匹配，以及受限的上下文长度。为解决这些问题，我们提出了RAVEN模型，该模型结合了检索增强掩码语言建模与前缀语言建模。我们进一步引入了“融合式上下文学习”机制，使模型能够利用更多上下文中的示例，且无需额外训练，从而提升少样本性能。通过大量实验，我们证明了这种简单而有效的设计显著提升了性能，尽管参数量大幅减少，但在某些场景下仍能取得与最先进语言模型相当的结果。我们的工作凸显了检索增强型编码器-解码器语言模型在上下文学习方面的潜力，并为该方向的后续研究提供了参考。
英文摘要：
In this paper, we investigate the in-context learning ability of retrieval-augmented encoder-decoder language models. We first conduct a comprehensive analysis of existing models and identify their limitations in in-context learning, primarily due to a mismatch between pretraining and inference, as well as a restricted context length. To address these issues, we propose RAVEN, a model that combines retrieval-augmented masked language modeling and prefix language modeling. We further introduce Fusion-in-Context Learning to enhance the few-shot performance by enabling the model to leverage more in-context examples without requiring additional training. Through extensive experiments, we demonstrate that our simple yet effective design significantly improves performance, achieving results comparable to the most advanced language models in certain scenarios, despite having substantially fewer parameters. Our work underscores the potential of retrieval-augmented encoder-decoder language models for in-context learning and encourages further research in this direction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
