arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.15399cs.LGcs.AIcs.CL
收藏

数据依赖的检索增强模型的统计框架

A Statistical Framework for Data-dependent Retrieval-Augmented Models

Google(谷歌公司)

Soumya Basu, Ankit Singh Rawat, Manzil Zaheer

AI总结:

本文提出了一个统计框架来研究包含检索器和预测器的数据依赖检索增强模型,提供了端到端训练方法并建立了超额风险界限,在开放域问答任务上验证了其有效性和统计分析结论。

摘要:

现代机器学习系统越来越多地用额外的相关信息来增强输入实例,以提升最终预测。尽管对此类检索增强模型的兴趣日益浓厚,但其基本属性和训练方法尚未得到充分理解。我们提出了一个统计框架来研究此类模型,该框架包含两个组件:1) 一个检索器,通过数据依赖的度量从大型语料库中识别相关信息;2) 一个预测器,它接收输入实例以及检索到的信息以做出最终预测。我们提出了一种原则性的方法来对这两个组件进行端到端训练,并与文献中的各种训练方法建立了联系。此外,我们建立了检索增强模型的超额风险界限,同时阐明了检索器和预测器对模型性能的贡献。我们在检索增强非常重要的开放域问答任务上,验证了我们提出的训练方法的实用性以及统计分析的关键结论。

英文摘要:

Modern ML systems increasingly augment input instances with additional relevant information to enhance final prediction. Despite growing interest in such retrieval-augmented models, their fundamental properties and training are not well understood. We propose a statistical framework to study such models with two components: 1) a {\em retriever} to identify the relevant information out of a large corpus via a data-dependent metric; and 2) a {\em predictor} that consumes the input instances along with the retrieved information to make the final predictions. We present a principled method for end-to-end training of both components and draw connections with various training approaches in the literature. Furthermore, we establish excess risk bounds for retrieval-augmented models while delineating the contributions of both retriever and predictor towards the model performance. We validate the utility of our proposed training methods along with the key takeaways from our statistical analysis on open domain question answering task where retrieval augmentation is important.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑