arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.17331cs.CLcs.LG
收藏

增强自适应检索器提升语言模型作为通用插件的泛化能力

Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In

Tsinghua University(清华大学);Microsoft Research(微软研究院);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心)

Zichun Yu, Chenyan Xiong, Shi Yu, Zhiyuan Liu

AI总结:

本文提出增强自适应检索器(AAR),通过从已知源语言模型学习偏好,使检索器能作为通用插件提升未知目标语言模型的零样本泛化能力,实验覆盖250M至175B参数模型并验证了偏好重叠性。

摘要:

检索增强可以通过向语言模型(LMs)提供外部信息来帮助其完成知识密集型任务。先前关于检索增强的工作通常联合微调检索器和语言模型,使二者紧密耦合。在本文中,我们探索了通用检索插件的方案:检索器用于辅助可能事先未知或无法一起微调的目标语言模型。为了给未见过的目标语言模型检索有用的文档,我们提出了增强自适应检索器(AAR),它学习从已知源语言模型获得的语言模型偏好。在MMLU和PopQA数据集上的实验表明,我们使用小型源语言模型训练的AAR能够显著提升更大目标语言模型的零样本泛化能力,这些目标语言模型规模从250M参数的Flan-T5到175B参数的InstructGPT。进一步分析表明,不同语言模型的偏好存在重叠,使得用单一源语言模型训练的AAR能够作为各种目标语言模型的通用插件。我们的代码已在https://github.com/OpenMatch/Augmentation-Adapted-Retriever开源。

英文摘要:

Retrieval augmentation can aid language models (LMs) in knowledge-intensive tasks by supplying them with external information. Prior works on retrieval augmentation usually jointly fine-tune the retriever and the LM, making them closely coupled. In this paper, we explore the scheme of generic retrieval plug-in: the retriever is to assist target LMs that may not be known beforehand or are unable to be fine-tuned together. To retrieve useful documents for unseen target LMs, we propose augmentation-adapted retriever (AAR), which learns LM's preferences obtained from a known source LM. Experiments on the MMLU and PopQA datasets demonstrate that our AAR trained with a small source LM is able to significantly improve the zero-shot generalization of larger target LMs ranging from 250M Flan-T5 to 175B InstructGPT. Further analysis indicates that the preferences of different LMs overlap, enabling AAR trained with a single source LM to serve as a generic plug-in for various target LMs. Our code is open-sourced at https://github.com/OpenMatch/Augmentation-Adapted-Retriever.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑