arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.03080cs.CLcs.AI
收藏

遗憾预训练:桥接先验与后验视角以增强知识基础

Regret Pre-training: Bridging Prior and Posterior Views for Enhanced Knowledge Grounding

University of Science and Technology of China(中国科学技术大学)

Mingkuan Zhao, Xiayu Sun, Wentao Hu, Suquan Chen, Jiaxuan Li, Xiaoyan Zhu, Xin Lai, Jiayin Wang

AI总结:

提出遗憾预训练框架,通过双视角架构利用未来信息增强因果语言模型,在OLMoE-1B-7B架构上平均准确率提升至33.9%。

摘要:

因果语言模型仅使用前文上下文对序列概率进行分解,在训练过程中尽管未来信息在训练数据中可用,但仍未被利用。本文介绍了遗憾预训练,这是一个基于学习使用特权信息范式的自监督框架。该框架采用双视角架构,其中单个模型同时生成因果学生分布和未来条件教师分布。训练目标通过遗憾损失增强标准语言建模,该损失最小化从教师到学生的KL散度,将未来感知信号传递到因果表示中。我们在OLMoE-1B-7B架构上研究了两种教师配置:LocalRegret,它将注意力扩展一个未来标记;以及GlobalRegret,它以目标位置被掩码的双向上下文为条件。在40亿个标记的训练后,对九个下游任务的实验表明,两种配置均持续优于基线。平均而言,GlobalRegret和LocalRegret分别达到33.9%和32.2%的准确率,超过了基线的30.2%。最值得注意的是,GlobalRegret将BoolQ性能提高了18.1个百分点(61.0%对42.9%)。该框架不引入额外参数,每个训练步骤仅需一次额外的推理模式前向传播。

英文摘要:

Causal language models factorize sequence probabilities using only preceding context, leaving future information unexploited during training despite its availability in the training data. This paper introduces Regret Pre-training, a self-supervised framework grounded in the Learning Using Privileged Information (LUPI) paradigm. The framework employs a dual-view architecture in which a single model generates both a causal Student distribution and a future-conditioned Teacher distribution. The training objective augments standard language modeling with a regret loss that minimizes the KL divergence from teacher to student, transferring future-aware signals to the causal representations. We investigate two teacher configurations on the OLMoE-1B-7B architecture:LocalRegret, which extends attention by one future token, andGlobalRegret, which conditions on bidirectional context with the target position masked. Experiments on nine downstream tasks following 4 billion tokens of training demonstrate that both configurations consistently outperform the baseline. On average,GlobalRegret andLocalRegret achieve 33.9% and 32.2% accuracy respectively, surpassing the baseline's 30.2%. Most notably,GlobalRegret improves BoolQ performance by 18.1 percentage points (61.0% vs 42.9%). The framework introduces no additional parameters and requires only one extra inference-mode forward pass per training step.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑