arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.06825cs.LGcs.AIcs.CV
收藏

AEGPO:适应性熵引导策略优化用于扩散模型

AEGPO: Adaptive Entropy-Guided Policy Optimization for Diffusion Models

Peking University(北京大学);Kling Team, Kuaishou Technology(快手科技 Kling 团队);The University of Hong Kong(香港大学);Beijing Normal University(北京师范大学)

Yuming Li, Qingyu Li, Chengyu Bai, Xiangyang Luo, Zeyue Xue, Wenyu Qin, Meng Wang, Yikai Wang, Shanghang Zhang

AI总结:

AEGPO通过双信号双层自适应优化提升扩散模型策略优化效率,实现更高效的收敛和更好的对齐性能。

摘要:

基于人类反馈的强化学习(RLHF)在对齐扩散模型和流模型方面显示出潜力,但像GRPO这样的策略优化方法却因低效和静态的采样策略而受到限制。这些方法将所有提示和去噪步骤一视同仁,忽略了样本学习价值的显著差异以及关键探索时刻的动态性质。为了解决这个问题,我们对GRPO训练过程中的内部注意力动态进行了详细分析,并发现了一个关键见解:注意力熵可以作为强大的双信号代理。首先,在不同的样本之间,注意力熵的相对变化(ΔEntropy),反映了当前策略与基础策略之间的分歧,是样本学习价值的稳健指标。其次,在去噪过程中,绝对注意力熵(Entropy(t))的峰值,量化了注意力分散程度,有效识别了高价值探索发生的关键时间点。基于这一观察,我们提出了适应性熵引导策略优化(AEGPO),一种新的双信号、双层自适应优化策略。在全局层面,AEGPO使用ΔEntropy动态分配滚出预算,优先处理学习价值更高的提示。在局部层面,它利用Entropy(t)的峰值,引导在关键高分散时间点进行选择性探索,而不是在所有去噪步骤中均匀进行。通过将计算集中在最有信息量的样本和最关键的时间点上,AEGPO实现了更高效和有效的策略优化。在文本到图像生成任务中的实验表明,AEGPO比标准GRPO变体显著加快了收敛速度,并在对齐性能上取得了更优的结果。

英文摘要:

Reinforcement learning from human feedback (RLHF) shows promise for aligning diffusion and flow models, yet policy optimization methods such as GRPO suffer from inefficient and static sampling strategies. These methods treat all prompts and denoising steps uniformly, ignoring substantial variations in sample learning value as well as the dynamic nature of critical exploration moments. To address this issue, we conduct a detailed analysis of the internal attention dynamics during GRPO training and uncover a key insight: attention entropy can serve as a powerful dual-signal proxy. First, across different samples, the relative change in attention entropy (ΔEntropy), which reflects the divergence between the current policy and the base policy, acts as a robust indicator of sample learning value. Second, during the denoising process, the peaks of absolute attention entropy (Entropy(t)), which quantify attention dispersion, effectively identify critical timesteps where high-value exploration occurs. Building on this observation, we propose Adaptive Entropy-Guided Policy Optimization (AEGPO), a novel dual-signal, dual-level adaptive optimization strategy. At the global level, AEGPO uses ΔEntropy to dynamically allocate rollout budgets, prioritizing prompts with higher learning value. At the local level, it exploits the peaks of Entropy(t) to guide exploration selectively at critical high-dispersion timesteps rather than uniformly across all denoising steps. By focusing computation on the most informative samples and the most critical moments, AEGPO enables more efficient and effective policy optimization. Experiments on text-to-image generation tasks demonstrate that AEGPO significantly accelerates convergence and achieves superior alignment performance compared to standard GRPO variants.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑