Memory Dial: 一种用于语言模型可控记忆的训练框架
Memory Dial: A Training Framework for Controllable Memorization in Language Models
AI总结:
Memory Dial通过调节记忆压力参数α,控制语言模型的记忆行为,实验显示α能有效调节记忆压力,大模型更敏感,常见序列更易记忆。
摘要:
语言模型中的记忆现象被广泛研究,但难以隔离和控制。理解模型何时何地记忆对解释预测至关重要,但现有方法是事后检测。我们引入Memory Dial训练框架,使记忆压力成为可控变量。通过参数α插值标准交叉熵与温度锐化目标,生成一系列在架构和训练设置相同(每轮扫瞄内)但记忆压力不同的模型。实验显示:(1) α可靠控制记忆压力,已见示例准确率单调增加而未见准确率稳定;(2) 大模型对记忆压力更敏感;(3) 常见序列比罕见序列更易记忆。进一步分析显示,效果在不同锐化温度下稳健,与单温度交叉熵不同,适用于多语言设置,并能检测自然单出现序列。Memory Dial为研究记忆行为如何产生及与泛化相互作用提供了受控实验框架。
英文摘要:
Memorization in language models is widely studied but remains difficult to isolate and control. Understanding when and what models memorize is essential for explaining their predictions, yet existing approaches are post-hoc: they can detect memorization in trained models, but cannot disentangle its effects from architecture, data, or optimization. We introduce Memory Dial, a training framework that makes memorization pressure an explicit, controllable variable. Memory Dial interpolates between standard cross-entropy and a temperature-sharpened objective via a single parameter $α$, producing a family of models identical in architecture and training setup (within each sweep), differing only in memorization pressure. Experiments across six architectures and five benchmarks demonstrate that: (1) $α$ reliably controls memorization pressure, with seen-example accuracy increasing monotonically while unseen accuracy remains stable; (2) larger models are more responsive to memorization pressure; and (3) frequent sequences are easier to memorize than rare ones. Additional analyses show that the effect is robust across a range of sharpening temperatures, differs qualitatively from single-temperature cross-entropy, transfers to multilingual settings, and is detectable even on naturally occurring single-occurrence sequences. Memory Dial provides a controlled experimental framework for studying how memorization behavior emerges and interacts with generalization in language models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。