通过数据重要性学习改进检索增强的大语言模型
Improving Retrieval-Augmented Large Language Models via Data Importance Learning
AI总结：
本文提出基于多线性扩展的数据重要性评估算法及其高效近似算法，通过仅剪枝或重加权检索语料库而无需进一步训练即可提升检索增强大语言模型性能，并验证了其高效性与有效性。
摘要：
检索增强使大语言模型能够利用外部知识，例如在问答和数据补全等任务中。然而，此类检索增强模型的性能受限于其底层检索语料库的数据质量。在本文中，我们提出了一种基于多线性扩展的算法，用于评估所检索数据点的数据重要性。多线性扩展中存在指数级数量的项，本文的一个关键贡献是提出了一种多项式时间算法，该算法在给定具有可加效用函数的检索增强模型和一个验证集的情况下，利用模型效用函数的多线性扩展精确计算检索语料库中数据点的数据重要性。我们进一步提出了一个更高效的(ε, δ)-近似算法。我们的实验结果表明，仅通过剪枝或重新加权检索语料库，无需进一步训练，即可提升大语言模型的性能。对于某些任务，这甚至使得一个小模型（例如GPT-JT）在结合搜索引擎API增强后，能够超越GPT-3.5（无检索增强）。此外，我们表明基于多线性扩展的权重在实践中可以高效计算（例如，对于一个包含1亿个元素的语料库，计算时间不到十分钟）。
英文摘要：
Retrieval augmentation enables large language models to take advantage of external knowledge, for example on tasks like question answering and data imputation. However, the performance of such retrieval-augmented models is limited by the data quality of their underlying retrieval corpus. In this paper, we propose an algorithm based on multilinear extension for evaluating the data importance of retrieved data points. There are exponentially many terms in the multilinear extension, and one key contribution of this paper is a polynomial time algorithm that computes exactly, given a retrieval-augmented model with an additive utility function and a validation set, the data importance of data points in the retrieval corpus using the multilinear extension of the model's utility function. We further proposed an even more efficient (ε, δ)-approximation algorithm. Our experimental results illustrate that we can enhance the performance of large language models by only pruning or reweighting the retrieval corpus, without requiring further training. For some tasks, this even allows a small model (e.g., GPT-JT), augmented with a search engine API, to outperform GPT-3.5 (without retrieval augmentation). Moreover, we show that weights based on multilinear extension can be computed efficiently in practice (e.g., in less than ten minutes for a corpus with 100 million elements).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
