arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.02178cs.CL
收藏

AR-MAP:自回归大语言模型是否是扩散大语言模型的隐式教师?

AR-MAP: Are Autoregressive Large Language Models Implicit Teachers for Diffusion Large Language Models?

AMAP, Alibaba Group(AMAP,阿里巴巴集团);Nanyang Technological University(南洋理工大学)

Liang Lin, Feng Xiong, Zengbin Wang, Kun Wang, Junhao Dong, Xuecai Hu, Yong Wang, Xiangxiang Chu

AI总结:

AR-MAP通过利用自回归大语言模型作为隐式教师,有效提升扩散大语言模型的偏好对齐性能。

摘要:

扩散大语言模型(DLLMs)已逐渐成为自回归模型的一种强大替代方案,能够实现多个位置上的并行标记生成。然而,由于证据下界(ELBO)基于似然估计引入的高方差,DLLMs的偏好对齐仍然具有挑战性。在本工作中,我们提出了一种新颖的迁移学习框架AR-MAP,该框架利用偏好对齐的自回归大语言模型(AR-LLMs)作为DLLM对齐的隐式教师。我们揭示了DLLMs可以通过简单的权重缩放有效地吸收来自AR-LLMs的对齐知识,利用这两种不同生成范式之间共享的架构结构。关键的是,我们的方法规避了直接DLLM对齐的高方差和计算开销。在多样化的偏好对齐任务上进行的全面实验表明,AR-MAP在所有任务和模型上取得了竞争性或更优的性能,平均得分为69.08%。我们的代码可在https://github.com/AMAP-ML/AR-MAP上获得。

英文摘要:

Diffusion Large Language Models (DLLMs) have emerged as a powerful alternative to autoregressive models, enabling parallel token generation across multiple positions. However, preference alignment of DLLMs remains challenging due to high variance introduced by Evidence Lower Bound (ELBO)-based likelihood estimation. In this work, we propose AR-MAP, a novel transfer learning framework that leverages preference-aligned autoregressive LLMs (AR-LLMs) as implicit teachers for DLLM alignment. We reveal that DLLMs can effectively absorb alignment knowledge from AR-LLMs through simple weight scaling, exploiting the shared architectural structure between these divergent generation paradigms. Crucially, our approach circumvents the high variance and computational overhead of direct DLLM alignment and comprehensive experiments across diverse preference alignment tasks demonstrate that AR-MAP achieves competitive or superior performance compared to existing DLLM-specific alignment methods, achieving 69.08\% average score across all tasks and models. Our Code is available at https://github.com/AMAP-ML/AR-MAP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑