自回归语言模型中的反转不变性
Reversal Invariance in Autoregressive Language Models
AI总结:
本文形式化了自回归语言建模目标的反转不变性特性,指出该对称性是当前预训练目标的局限,并呼吁未来研究应通过时间不对称性视角设计能明确建模语言箭头的损失函数与架构。
摘要:
我们形式化了因果(自回归)语言建模(CLM)目标的一个结构特性:反转不变性。形式上,下一token预测损失为语料库及其反转分配相同的似然,这意味着标准CLM预训练是方向盲的。这种对称性解释了为何在反转文本上训练的模型能取得与在正向文本上训练的模型相当的性能,尽管人类语言和推理具有固有的时间不对称性。我们认为,这种不变性代表了当前预训练目标的局限性,而非良性伪影。如果自然语言编码了方向性依赖——音韵、形态或因果——对称目标可能无法捕获它们。因此,我们提出通过时间不对称性的视角审视预训练,激励未来在损失函数和架构上开展明确建模语言箭头同时保留标准语言建模能力的工作。
英文摘要:
We formalize a structural property of the causal (autoregressive) language modeling (CLM) objective: reversal invariance. Formally, the next-token prediction loss assigns identical likelihood to a corpus and its reversal, implying that standard CLM pretraining is direction-blind. This symmetry explains why models trained on reversed text can achieve comparable performance to those trained on forward text, despite the inherently time-asymmetric nature of human language and reasoning. We argue that this invariance represents a limitation of current pretraining objectives rather than a benign artifact. If natural language encodes directional dependencies - phonological, morphological, or causal - a symmetric objective may fail to capture them. We therefore propose viewing pretraining through the lens of temporal asymmetry, motivating future work on loss functions and architectures that explicitly model the arrow of language while retaining standard language modeling capacity.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。