面向可靠大语言模型推理的自动课程专家迭代方法
Automatic Curriculum Expert Iteration for Reliable LLM Reasoning
AI总结：
针对大语言模型推理中的幻觉与惰性问题，提出自动课程专家迭代方法Auto-CEI，通过专家迭代修正推理路径、课程机制调整奖励，平衡作答主动性与保守性，在多类推理任务上实现更优的能力对齐效果。
摘要：
幻觉（即生成看似合理但不准确的内容）和惰性（即过度拒绝或默认回答“我不知道”）始终是大语言模型（LLM）推理领域的主要挑战。当前减少幻觉的研究主要聚焦于知识 grounding 任务中的事实错误，往往忽略了与错误推理相关的幻觉问题。同时，部分方法会让LLM变得过于保守，限制了其问题解决能力。为缓解推理任务中的幻觉与惰性问题，我们提出自动课程专家迭代方法（Automatic Curriculum Expert Iteration，Auto-CEI），用于增强LLM推理能力并使响应与模型能力对齐——在能力范围内主动作答，在任务超出能力时予以拒绝。在我们的方法中，专家迭代（Expert Iteration）探索LLM策略附近的推理轨迹，将错误路径引导回正轨，以减少累积误差并提升鲁棒性；它还会在充分的推理尝试后推动模型给出恰当的“我不知道”响应。课程机制自动调整奖励，激励模型在承认无法完成之前进行更充分的推理，从而拓展LLM的推理边界，并使其行为与这些边界对齐。我们在逻辑推理、数学和规划任务上将Auto-CEI与多种SOTA基线进行对比，Auto-CEI通过有效平衡主动性与保守性，实现了更优的对齐效果。代码可在https://github.com/SalesforceAIResearch/Auto-CEI 获取。
英文摘要：
Hallucinations (i.e., generating plausible but inaccurate content) and laziness (i.e. excessive refusals or defaulting to "I don't know") persist as major challenges in LLM reasoning. Current efforts to reduce hallucinations primarily focus on factual errors in knowledge-grounded tasks, often neglecting hallucinations related to faulty reasoning. Meanwhile, some approaches render LLMs overly conservative, limiting their problem-solving capabilities. To mitigate hallucination and laziness in reasoning tasks, we propose Automatic Curriculum Expert Iteration (Auto-CEI) to enhance LLM reasoning and align responses to the model's capabilities--assertively answering within its limits and declining when tasks exceed them. In our method, Expert Iteration explores the reasoning trajectories near the LLM policy, guiding incorrect paths back on track to reduce compounding errors and improve robustness; it also promotes appropriate "I don't know" responses after sufficient reasoning attempts. The curriculum automatically adjusts rewards, incentivizing extended reasoning before acknowledging incapability, thereby pushing the limits of LLM reasoning and aligning its behaviour with these limits. We compare Auto-CEI with various SOTA baselines across logical reasoning, mathematics, and planning tasks, where Auto-CEI achieves superior alignment by effectively balancing assertiveness and conservativeness. The code is available at https://github.com/SalesforceAIResearch/Auto-CEI .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
