ARES：通过多样化AI反馈增强多模态思维链推理的交替强化学习与监督微调方法
ARES: Alternating Reinforcement Learning and Supervised Fine-Tuning for Enhanced Multi-Modal Chain-of-Thought Reasoning Through Diverse AI Feedback
AI总结：
针对大语言模型现有反馈优化方法粒度粗的问题，提出两阶段算法ARES，交替结合句子级AI反馈的强化学习与错误修正的监督微调，提升多模态思维链推理效果与答案准确率。
摘要：
大语言模型（LMMs）擅长理解人类指令，在广泛任务中展现出出色效果。基于人类反馈的强化学习（RLHF）和基于AI反馈的强化学习（RLAIF）通过让模型对齐特定偏好，进一步优化大语言模型（LLMs）。这些方法主要对完整生成内容使用基于排序的反馈。借助GPT-4、Claude 3 Opus等先进AI模型（教师模型），我们可以获取多种人类难以提供的详细反馈。我们提出两阶段算法ARES，即交替强化学习（RL）与监督微调（SFT）算法。第一阶段，我们请求教师模型对思维链（CoT）中每句话对解题的贡献度打分。这种句子级反馈使我们能关注单个有价值片段，为RL流程提供更细粒度的奖励。第二阶段，我们在RL阶段后请教师模型修正错误推理。RL流程需要大量超参数调优工作，且常生成重复词汇、句子不完整等错误。借助修正反馈，我们通过SFT稳定RL微调后的模型。我们在多模态数据集ScienceQA和A-OKVQA上开展实验，验证所提方法的有效性。经GPT-4o判定，ARES的推理解释相对于基线模型的胜率约为70%。此外我们观察到，推理解释的提升使多模态数据集上的推理答案准确率平均提高2.5%。
英文摘要：
Large Multimodal Models (LMMs) excel at comprehending human instructions and demonstrate remarkable results across a broad spectrum of tasks. Reinforcement Learning from Human Feedback (RLHF) and AI Feedback (RLAIF) further refine LLMs by aligning them with specific preferences. These methods primarily use ranking-based feedback for entire generations. With advanced AI models (Teacher), such as GPT-4 and Claude 3 Opus, we can request various types of detailed feedback that are expensive for humans to provide. We propose a two-stage algorithm ARES that Alternates REinforcement Learning (RL) and Supervised Fine-Tuning (SFT). First, we request the Teacher to score how much each sentence contributes to solving the problem in a Chain-of-Thought (CoT). This sentence-level feedback allows us to consider individual valuable segments, providing more granular rewards for the RL procedure. Second, we ask the Teacher to correct the wrong reasoning after the RL stage. The RL procedure requires massive efforts for hyperparameter tuning and often generates errors like repetitive words and incomplete sentences. With the correction feedback, we stabilize the RL fine-tuned model through SFT. We conduct experiments on multi-model dataset ScienceQA and A-OKVQA to demonstrate the effectiveness of our proposal. ARES rationale reasoning achieves around 70% win rate against baseline models judged by GPT-4o. Additionally, we observe that the improved rationale reasoning leads to a 2.5% increase in inference answer accuracy on average for the multi-modal datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
