利用人类与人工智能反馈的强化学习增强大语言模型的物理问题求解能力
Enhancing LLMs for Physics Problem-Solving using Reinforcement Learning with Human-AI Feedback
AI总结：
本文提出RLHAIF方法，结合人类与AI反馈强化学习，并在PhyQA上评估PPO、DPO和Remax，显著提升LLaMA2与Mistral的物理推理及答题准确性。
摘要：
大语言模型（LLMs）已在基于文本的任务中展现出强大能力，但在应对物理问题所需的复杂推理时仍存在困难，尤其是在高级算术和概念理解方面。尽管已有研究探索利用提示工程和检索增强生成（RAG）等技术提升LLMs在物理教育中的表现，但在解决其物理推理局限性方面投入仍不足。本文提出一种新颖方法，通过人类与人工智能反馈强化学习（RLHAIF）提升LLMs在物理问题上的表现。我们评估了多种强化学习方法，包括近端策略优化（PPO）、直接偏好优化（DPO）和Remax优化。选择这些方法是为了在PhyQA数据集上研究不同设置下的强化学习策略性能，该数据集包含来自高中教材的具有挑战性的物理问题。我们的RLHAIF模型在LLaMA2和Mistral等主流LLMs上进行测试，取得了优异结果，尤其是MISTRAL-PPO模型，在推理和准确性方面表现出显著提升。该模型获得高分，METEOR分数为58.67，推理分数为0.74，成为该领域未来物理推理研究的有力范例。
英文摘要：
Large Language Models (LLMs) have demonstrated strong capabilities in text-based tasks but struggle with the complex reasoning required for physics problems, particularly in advanced arithmetic and conceptual understanding. While some research has explored ways to enhance LLMs in physics education using techniques such as prompt engineering and Retrieval Augmentation Generation (RAG), not enough effort has been made in addressing their limitations in physics reasoning. This paper presents a novel approach to improving LLM performance on physics questions using Reinforcement Learning with Human and Artificial Intelligence Feedback (RLHAIF). We evaluate several reinforcement learning methods, including Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), and Remax optimization. These methods are chosen to investigate RL policy performance with different settings on the PhyQA dataset, which includes challenging physics problems from high school textbooks. Our RLHAIF model, tested on leading LLMs like LLaMA2 and Mistral, achieved superior results, notably with the MISTRAL-PPO model, demonstrating marked improvements in reasoning and accuracy. It achieved high scores, with a 58.67 METEOR score and a 0.74 Reasoning score, making it a strong example for future physics reasoning research in this area.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
