分布对齐博弈用于答案级微调
Distributional Alignment Games for Answer-Level Fine-Tuning
AI总结：
本文提出分布对齐博弈框架，通过策略与目标的博弈达到答案级优化，统一了多样性与自改进方法，提升数学推理任务效率。
摘要：
我们聚焦于答案级微调（ALFT）问题，目标是基于最终答案的正确性或属性优化语言模型，而非生成过程中的具体推理路径。直接优化答案级目标因需边际化庞大的潜在推理路径空间而计算上不可行。为此，我们提出一个通用博弈论框架，将问题提升为分布对齐博弈。我们将ALFT表述为策略（生成器）与目标（辅助分布）之间的两 player 游戏。我们证明该游戏的纳什均衡恰好对应原始答案级优化问题的解。这种变分视角将不可行的边际化问题转化为可处理的投影问题。我们展示该框架统一了最近的多样性与自改进（一致性）方法，并提供与群体相对策略优化（GRPO）兼容的高效算法，如一致性-GRPO，从而在数学推理任务中实现显著的复杂性提升。
英文摘要：
We focus on the problem of \emph{Answer-Level Fine-Tuning} (ALFT), where the goal is to optimize a language model based on the correctness or properties of its final answers, rather than the specific reasoning traces used to produce them. Directly optimizing answer-level objectives is computationally intractable due to the need to marginalize over the vast space of latent reasoning paths. To overcome this, we propose a general game-theoretical framework that lifts the problem to a \emph{Distributional Alignment Game}. We formulate ALFT as a two-player game between a Policy (the generator) and a Target (an auxiliary distribution). We prove that the Nash Equilibrium of this game corresponds exactly to the solution of the original answer-level optimization problem. This variational perspective transforms the intractable marginalization problem into a tractable projection problem. We demonstrate that this framework unifies recent approaches to diversity and self-improvement (coherence) and provide efficient algorithms compatible with Group Relative Policy Optimization (GRPO), such as Coherence-GRPO, yielding significant complexity gains in mathematical reasoning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
