arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.15076cs.CL
收藏

语言模型的元学习在线自适应

Meta-Learning Online Adaptation of Language Models

Stanford University(斯坦福大学)

Nathan Hu, Eric Mitchell, Christopher D. Manning, Chelsea Finn

AI总结:

本文提出CaMeLS方法,通过元训练小型自回归模型动态重新加权在线微调中的token损失,显著提升了语言模型对文档流的信息吸收率,解决了静态模型知识过时问题。

摘要:

大型语言模型在其参数中编码了令人印象深刻的广泛世界知识。然而,静态语言模型中的知识会过时,限制了模型的有效“保质期”。虽然在线微调可以减少这种退化,但我们发现,在文档流上简单地进行微调会导致信息吸收率较低。我们假设在线微调没有充分关注重要信息。也就是说,代表事实信息的重要token的梯度信号被固有噪声token的梯度所淹没,这表明动态的、上下文感知的学习率可能是有益的。因此,我们提出学习对哪些token进行加权。我们元训练一个小型自回归模型,在在线微调期间为每个token重新加权语言建模损失,目标是在单个加权梯度步骤后,最大化过时的基线问答模型回答关于文档问题的能力。我们将这种方法称为Context-aware Meta-learned Loss Scaling (CaMeLS)。在三种不同的文档分布中,我们的实验发现,与标准微调和重新加权token损失的基线启发式方法相比,CaMeLS在数千个文档流上提供了显著改善的信息吸收率。

英文摘要:

Large language models encode impressively broad world knowledge in their parameters. However, the knowledge in static language models falls out of date, limiting the model's effective "shelf life." While online fine-tuning can reduce this degradation, we find that naively fine-tuning on a stream of documents leads to a low level of information uptake. We hypothesize that online fine-tuning does not sufficiently attend to important information. That is, the gradient signal from important tokens representing factual information is drowned out by the gradient from inherently noisy tokens, suggesting that a dynamic, context-aware learning rate may be beneficial. We therefore propose learning which tokens to upweight. We meta-train a small, autoregressive model to reweight the language modeling loss for each token during online fine-tuning, with the objective of maximizing the out-of-date base question-answering model's ability to answer questions about a document after a single weighted gradient step. We call this approach Context-aware Meta-learned Loss Scaling (CaMeLS). Across three different distributions of documents, our experiments find that CaMeLS provides substantially improved information uptake on streams of thousands of documents compared with standard fine-tuning and baseline heuristics for reweighting token losses.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑