arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.13466cs.CLcs.AIcs.LG
收藏

语言模型记忆与记忆模型用于语言

Language Model Memory and Memory Models for Language

IBM(IBM公司)

Benjamin L. Badger

AI总结:

研究探讨了语言模型和记忆模型在信息存储中的能力差异,发现语言模型的嵌入向量信息较少,而自编码器在输入再生训练中能形成接近完美的记忆,提出了一种可并行的编码器-解码器记忆模型架构,并通过结合因果和信息保留目标函数来提升记忆形成和解码能力。

摘要:

机器学习模型存储输入信息的能力,类似于“记忆”的概念,在隐藏层向量嵌入中被广泛使用但未充分表征。我们发现,无论数据和计算规模如何,语言模型嵌入通常包含相对较少的输入信息。相比之下,用于输入再生训练的自编码器嵌入能够形成几乎完美的记忆。用记忆嵌入替代令牌序列可带来显著的计算效率,从而引入一种可并行的编码器-解码器记忆模型架构。在因果训练后,这些模型包含信息贫乏的嵌入,无法进行任意信息访问,但通过结合因果和信息保留目标函数,它们学会形成和解码信息丰富的记忆。通过冻结高保真编码器并采用课程训练方法,解码器首先学习处理记忆,然后学习预测下一个令牌。我们引入了观点,即仅使用下一个令牌预测训练不足以准确形成记忆,因为目标本身不可逆,从而推动在输入不完全暴露的情况下使用结合目标函数的模型。

英文摘要:

The ability of machine learning models to store input information in hidden layer vector embeddings, analogous to the concept of `memory', is widely employed but not well characterized. We find that language model embeddings typically contain relatively little input information regardless of data and compute scale during training. In contrast, embeddings from autoencoders trained for input regeneration are capable of nearly perfect memory formation. The substitution of memory embeddings for token sequences leads to substantial computational efficiencies, motivating the introduction of a parallelizable encoder-decoder memory model architecture. Upon causal training these models contain information-poor embeddings incapable of arbitrary information access, but by combining causal and information retention objective functions they learn to form and decode information-rich memories. Training can be further streamlined by freezing a high fidelity encoder followed by a curriculum training approach where decoders first learn to process memories and then learn to additionally predict next tokens. We introduce the perspective that next token prediction training alone is poorly suited for accurate memory formation as the objective itself is non-invertible, motivating the use of combined objective functions for models where the entire input is not exposed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑