PowerFlow: 通过原则性分布匹配释放LLMs的双重特性
PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching
AI总结：
提出PowerFlow框架，将无监督微调重构成分布匹配问题，利用GFlowNet和长度感知轨迹平衡目标，通过调整α-幂分布方向性激发LLMs的逻辑推理或创造性。
摘要：
无监督内部反馈强化学习（RLIF）已成为一种有前景的范式，可以在没有外部监督的情况下激发大型语言模型（LLMs）的潜在能力。然而，当前方法依赖于启发式内在奖励，通常缺乏明确的理论优化目标，并且容易产生退化偏差。在这项工作中，我们引入了PowerFlow，一个原则性框架，将无监督微调重新表述为分布匹配问题。通过将GFlowNet视为未归一化密度的摊销变分采样器，我们提出了一个长度感知的轨迹平衡目标，明确抵消了自回归生成中固有的结构长度偏差。通过针对$α$-幂分布，PowerFlow能够方向性地激发LLMs的双重特性：锐化分布（$α> 1$）以增强逻辑推理，或展平分布（$α< 1$）以释放表达性创造力。大量实验表明，PowerFlow始终优于现有的RLIF方法，匹配甚至超过有监督的GRPO。此外，通过减轻对齐模型中的过度锐化，我们的方法在多样性和质量上同时取得提升，在创造性任务中推动了帕累托前沿。
英文摘要：
Unsupervised Reinforcement Learning from Internal Feedback (RLIF) has emerged as a promising paradigm for eliciting the latent capabilities of Large Language Models (LLMs) without external supervision. However, current methods rely on heuristic intrinsic rewards, which often lack a well-defined theoretical optimization target and are prone to degenerative biases. In this work, we introduce PowerFlow, a principled framework that reformulates unsupervised fine-tuning as a distribution matching problem. By casting GFlowNet as an amortized variational sampler for unnormalized densities, we propose a length-aware Trajectory-Balance objective that explicitly neutralizes the structural length biases inherent in autoregressive generation. By targeting $α$-power distributions, PowerFlow enables the directional elicitation of the dual nature of LLMs: sharpening the distribution ($α> 1$) to intensify logical reasoning, or flattening it ($α< 1$) to unlock expressive creativity. Extensive experiments demonstrate that PowerFlow consistently outperforms existing RLIF methods, matching or even exceeding supervised GRPO. Furthermore, by mitigating over-sharpening in aligned models, our approach achieves simultaneous gains in diversity and quality, shifting the Pareto frontier in creative tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
