arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.07132stat.MLcs.LG
收藏

离散共轭匹配

Discrete Adjoint Matching

Massachusetts Institute of Technology(麻省理工学院);FAIR at Meta(Meta 的 FAIR)

Oswin So, Brian Karrer, Chuchu Fan, Ricky T. Q. Chen, Guan-Horng Liu

AI总结:

本文提出离散共轭匹配方法,用于微调离散生成模型,通过离散域上的共轭估计器解决熵正则化奖励优化问题。

摘要:

用于解决熵正则化奖励优化的计算方法——一类广泛用于生成模型微调的问题——已迅速发展。其中,共轭匹配(AM,Domingo-Enrich等人,2025)在具有可微奖励的连续状态空间中证明了高度有效性。然而,将这些实用成功转移到离散生成建模仍特别具有挑战性且几乎未被探索,主要是由于生成模型类别从连续状态空间急剧转移到离散状态空间,而后者无处不可微。在本文中,我们提出了离散共轭匹配(DAM)——一种用于微调离散生成模型的AM离散变体,这些模型由连续时间马尔可夫链特征化,例如基于扩散的大型语言模型。DAM的核心是引入离散共轭——一种估计原始问题最优解的估计器,但是在离散域上构建的,从而可以应用标准匹配框架。这是通过纯粹的统计观点得出的,与AM中的控制理论观点形成对比,从而为通用共轭估计器开辟了新的算法机会。我们展示了DAM在合成和数学推理任务上的有效性。

英文摘要:

Computation methods for solving entropy-regularized reward optimization -- a class of problems widely used for fine-tuning generative models -- have advanced rapidly. Among those, Adjoint Matching (AM, Domingo-Enrich et al., 2025) has proven highly effective in continuous state spaces with differentiable rewards. Transferring these practical successes to discrete generative modeling, however, remains particularly challenging and largely unexplored, mainly due to the drastic shift in generative model classes to discrete state spaces, which are nowhere differentiable. In this work, we propose Discrete Adjoint Matching (DAM) -- a discrete variant of AM for fine-tuning discrete generative models characterized by Continuous-Time Markov Chains, such as diffusion-based large language models. The core of DAM is the introduction of discrete adjoint-an estimator of the optimal solution to the original problem but formulated on discrete domains-from which standard matching frameworks can be applied. This is derived via a purely statistical standpoint, in contrast to the control-theoretic viewpoint in AM, thereby opening up new algorithmic opportunities for general adjoint-based estimators. We showcase DAM's effectiveness on synthetic and mathematical reasoning tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑