通过想象、搜索与批判实现大语言模型的自我提升
Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing
AI总结:
针对大语言模型复杂推理能力受限且依赖标注数据的问题,提出融合蒙特卡洛树搜索的AlphaLLM框架,通过提示合成、适配语言任务的搜索和多批判模型构建自提升循环,无额外标注即可显著增强数学推理性能。
摘要:
尽管大语言模型(LLMs)在各类任务上展现出令人瞩目的能力,但它们在涉及复杂推理与规划的场景中仍存在不足。近期研究提出了先进的提示工程技术,以及使用高质量数据进行微调以增强LLMs推理能力的必要性,但这些方法本质上受限于数据的可得性与质量。鉴于此,自我纠错与自我学习成为可行的解决方案,这类策略让LLMs能够优化自身输出,并从自我评估的奖励中学习。然而,LLMs自我优化回复的效果,尤其是在复杂推理与规划任务中的效果,仍存疑问。\n本文提出了用于LLMs自我提升的AlphaLLM,该方法将蒙特卡洛树搜索(MCTS)与LLMs结合,构建自我提升循环,从而在无需额外标注的情况下增强LLMs的能力。受AlphaGo成功的启发,AlphaLLM解决了将MCTS与LLM结合用于自我提升的独特挑战,包括数据稀缺、语言任务搜索空间过大以及语言任务反馈的主观性问题。AlphaLLM由提示合成组件、专为语言任务定制的高效MCTS方法,以及用于提供精准反馈的三个批判模型组成。我们在数学推理任务上的实验结果表明,AlphaLLM在无需额外标注的情况下显著提升了LLMs的性能,展现了LLMs自我提升的潜力。
英文摘要:
Despite the impressive capabilities of Large Language Models (LLMs) on various tasks, they still struggle with scenarios that involves complex reasoning and planning. Recent work proposed advanced prompting techniques and the necessity of fine-tuning with high-quality data to augment LLMs' reasoning abilities. However, these approaches are inherently constrained by data availability and quality. In light of this, self-correction and self-learning emerge as viable solutions, employing strategies that allow LLMs to refine their outputs and learn from self-assessed rewards. Yet, the efficacy of LLMs in self-refining its response, particularly in complex reasoning and planning task, remains dubious. In this paper, we introduce AlphaLLM for the self-improvements of LLMs, which integrates Monte Carlo Tree Search (MCTS) with LLMs to establish a self-improving loop, thereby enhancing the capabilities of LLMs without additional annotations. Drawing inspiration from the success of AlphaGo, AlphaLLM addresses the unique challenges of combining MCTS with LLM for self-improvement, including data scarcity, the vastness search spaces of language tasks, and the subjective nature of feedback in language tasks. AlphaLLM is comprised of prompt synthesis component, an efficient MCTS approach tailored for language tasks, and a trio of critic models for precise feedback. Our experimental results in mathematical reasoning tasks demonstrate that AlphaLLM significantly enhances the performance of LLMs without additional annotations, showing the potential for self-improvement in LLMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。