arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.08020cs.LGcs.AI
收藏

在测试时高效学习:大语言模型的主动微调

Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs

Jonas Hübotter, Sascha Bongni, Ido Hakimi, Andreas Krause

AI总结:

针对最近邻检索易选冗余数据的问题,本文提出主动数据选择算法 SIFT,通过优化信息增益提升测试时微调效果,并实现按性能增益自适应分配计算资源。

摘要:

近期针对语言模型微调的工作通常依赖自动数据选择,常见做法是从大规模数据集中进行最近邻检索。然而,我们从理论上证明,这种方法倾向于选择冗余数据,从而限制其效果,甚至可能损害性能。为解决这一问题,我们提出了 SIFT,一种旨在减少给定提示下模型响应不确定性的数据选择算法,该算法统一了检索与主动学习的思想。最近邻检索在存在信息重复的情况下通常会失效,而 SIFT 能够考虑信息重复,并优化所选样本的整体信息增益。我们将评估重点放在 Pile 数据集上针对特定提示的语言模型测试时微调,并表明 SIFT 在计算开销极小的情况下持续优于最近邻检索。此外,我们证明我们的不确定性估计可以预测测试时微调的性能提升,并据此开发了一种自适应算法,使测试时计算投入与已实现的性能增益成比例。我们提供了 activeft(主动微调)库,可作为最近邻检索的直接替代方案。

英文摘要:

Recent efforts in fine-tuning language models often rely on automatic data selection, commonly using Nearest Neighbors retrieval from large datasets. However, we theoretically show that this approach tends to select redundant data, limiting its effectiveness or even hurting performance. To address this, we introduce SIFT, a data selection algorithm designed to reduce uncertainty about the model's response given a prompt, which unifies ideas from retrieval and active learning. Whereas Nearest Neighbor retrieval typically fails in the presence of information duplication, SIFT accounts for information duplication and optimizes the overall information gain of the selected examples. We focus our evaluations on fine-tuning at test-time for prompt-specific language modeling on the Pile dataset, and show that SIFT consistently outperforms Nearest Neighbor retrieval, with minimal computational overhead. Moreover, we show that our uncertainty estimates can predict the performance gain of test-time fine-tuning, and use this to develop an adaptive algorithm that invests test-time compute proportional to realized performance gains. We provide the $\texttt{activeft}$ (Active Fine-Tuning) library which can be used as a drop-in replacement for Nearest Neighbor retrieval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑