基于反思反馈的强化学习（RLRF）：通过细粒度自我反思对齐并改进LLMs
Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflection
AI总结：
针对RLHF导致的表面化对齐问题，提出RLRF框架，利用细粒度自我反思机制探索和优化响应，并通过RL算法微调模型，有效提升了LLMs在多项推理任务上的核心能力。
摘要：
尽管RLHF在使LLMs与人类偏好对齐方面具有潜力，但它常导致表面化对齐，优先改变风格而非提升LLMs的下游性能。未充分明确的偏好会掩盖对齐模型的方向。缺乏探索则限制了识别理想输出以改进模型。为克服这些挑战，我们提出了一种新框架：基于反思反馈的强化学习（RLRF），它利用基于详细标准的细粒度反馈来提升LLMs的核心能力。RLRF采用自我反思机制系统地探索和优化LLM响应，随后通过RL算法结合有前景的响应来微调模型。我们在Just-Eval、Factuality和Mathematical Reasoning上的实验证明了RLRF在超越表面调整方面的有效性与变革潜力。
英文摘要：
Despite the promise of RLHF in aligning LLMs with human preferences, it often leads to superficial alignment, prioritizing stylistic changes over improving downstream performance of LLMs. Underspecified preferences could obscure directions to align the models. Lacking exploration restricts identification of desirable outputs to improve the models. To overcome these challenges, we propose a novel framework: Reinforcement Learning from Reflective Feedback (RLRF), which leverages fine-grained feedback based on detailed criteria to improve the core capabilities of LLMs. RLRF employs a self-reflection mechanism to systematically explore and refine LLM responses, then fine-tuning the models via a RL algorithm along with promising responses. Our experiments across Just-Eval, Factuality, and Mathematical Reasoning demonstrate the efficacy and transformative potential of RLRF beyond superficial surface-level adjustment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
