arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.21744cs.CL
收藏

时间指导用于大语言模型

Temporal Guidance for Large Language Models

College of Artificial Intelligence, MIIT Key Laboratory of Pattern Analysis(人工智能学院,模式分析与机器智能关键实验室);The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing University of Aeronautics(脑机智能技术关键实验室,教育部,南京航空航天大学)

Hong-Kai Zheng, Piji Li

AI总结:

本文提出时间指导策略,通过多令牌预测和轻量级投影器提升大语言模型生成质量,同时保持低计算开销。

摘要:

对比解码(CD)增强了大语言模型(LLMs)的生成质量,但由于需要辅助模型,导致了显著的额外计算开销。现有的内部自对比解码方法,如通过层对比解码(DoLa),专注于不同层之间的差异,这在小规模模型上尤为不稳定。在本工作中,基于LLMs表现出局部偏好的观察,我们提出了一种沿时间维度的新对比指导策略,即时间指导(TeGu)。我们的方法巧妙地利用多令牌预测(MTP)来构建较弱的业余预测以供模型自对比。为了标准化该机制的实现,我们进一步引入了一个轻量级的条件MTP投影器(cMTPP),这避免了其他MTP模块所要求的维护多个独立网络。在各种模型系列和基准测试中,TeGu在保持低额外内存消耗和计算开销的同时实现了显著的性能提升。

英文摘要:

Contrastive Decoding (CD) enhances the generation quality of large language models (LLMs) but incurs significant additional computational overhead due to the need for an auxiliary model. Existing internal self-contrastive decoding methods, such as Decoding by Contrasting Layers (DoLa), focus on discrepancies across different layers, which are notably unstable on small-scale models. In this work, based on the observation that LLMs exhibit local preferences, we propose a novel contrastive guidance strategy along the temporal dimension, namely Temporal Guidance (TeGu). Our method ingeniously leverages Multi-Token Prediction (MTP) to construct weaker amateur predictions for model self-contrast. To standardize the implementation of this mechanism, we further introduce a lightweight Conditional MTP Projector (cMTPP), which avoids maintaining multiple independent networks as required by other MTP modules. Across various model series and benchmarks, TeGu achieves significant performance improvements while maintaining low additional memory consumption and computational overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑