OPERA: 通过基于客观困惑度的强化学习对齐开放式推理
OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning
AI总结：
提出OPERA方法，利用困惑度动态作为内在奖励信号，结合冷启动数据合成和困惑度优先的推理轨迹选择，实现开放式任务中对齐，在Qwen3-8B上达到开源模型最优。
摘要：
强化学习（RL）使大语言模型在数学和代码生成等客观推理任务中表现出色。然而，将RL应用于开放式任务（如创意写作）仍然具有挑战性，因为LLM作为评判者的奖励模型常常表现出风格偏差和位置不一致，导致监督不稳定。为了解决这个问题，我们提出了OPERA（基于客观困惑度的反思对齐），该方法用源自困惑度动态的内在奖励取代不可靠的外部评判者。具体来说，我们从困惑度动态中推导出内在奖励信号，量化关键反思状态下的不确定性减少。在冷启动阶段，我们引入了一种数据合成方法，利用精心设计的引导词生成多样化的推理轨迹，以及利用内部对数概率识别逻辑一致推理分支的困惑度优先展开。该流程生成了一个包含20,000条高质量推理轨迹的大规模数据集。实证评估一致证明了我们的方法在开放式任务对齐中的可扩展性和有效性。在Qwen3-8B上实施OPERA建立了开源模型的新最先进水平，在某些开放式任务中达到或超过了Gemini2.5和MiniMax-M2.5等专有模型。代码可在该网址获取。
英文摘要：
Reinforcement Learning (RL) has enabled LLMs to excel in objective reasoning tasks such as mathematics and code generation. However, applying RL to open-ended tasks, such as creative writing, remains challenging because LLM-as-a-judge reward models often exhibit stylistic biases and positional inconsistencies, leading to unstable supervision. To address this, we propose OPERA (Objective Perplexity-based Reflective Alignment), which replaces unreliable external judges with intrinsic rewards derived from perplexity dynamics. Specifically, we derive an intrinsic reward signal from perplexity dynamics, quantifying uncertainty reduction at critical reflective states. During the cold-start phase, we introduce a data synthesis method that leverages carefully designed guiding words to generate diverse reasoning traces, along with perplexity-prioritized rollouts that utilize internal log-probabilities to identify logically consistent reasoning branches. This pipeline yields a large-scale dataset comprising 20,000 high-quality reasoning trajectories. Empirical evaluations consistently demonstrate the scalability and efficacy of our approach in alignment for open-ended tasks. Implementing OPERA on Qwen3-8B establishes a new state-of-the-art among open-source models, achieving parity with or surpassing proprietary models like Gemini2.5 and MiniMax-M2.5 in some open-ended tasks. The code is available at https://github.com/pangpang-xuan/OPERA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
