擦除以提升：用于搜索增强大语言模型的可擦除强化学习
Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMs
AI总结：
本文提出Erasable Reinforcement Learning框架，通过识别并擦除推理中的错误步骤，提升搜索增强大语言模型在多步推理中的鲁棒性，实验表明其在多个基准测试中取得显著提升。
摘要：
尽管搜索增强的大语言模型（LLMs）表现出色，但其在复杂多跳推理中的可靠性仍有限。这种限制源于三个根本性挑战：分解错误、检索缺失和推理错误。我们提出可擦除强化学习（ERL），一种将脆弱推理转化为稳健过程的新框架。ERL明确识别故障步骤，擦除它们，并在原地重新生成推理，防止缺陷逻辑传播。训练后的模型称为ESearch，在HotpotQA、MuSiQue、2Wiki和Bamboogle等基准测试中取得显著提升，3B模型在EM和F1上分别提高8.48%和11.56%，7B模型在EM和F1上分别提高5.38%和7.22%。这些发现表明，可擦除强化学习为LLMs的稳健多步推理提供了强大的范式转变。
英文摘要：
While search-augmented large language models (LLMs) exhibit impressive capabilities, their reliability in complex multi-hop reasoning remains limited. This limitation arises from three fundamental challenges: decomposition errors, where tasks are incorrectly broken down; retrieval missing, where key evidence fails to be retrieved; and reasoning errors, where flawed logic propagates through the reasoning chain. A single failure in any of these stages can derail the final answer. We propose Erasable Reinforcement Learning (ERL), a novel framework that transforms fragile reasoning into a robust process. ERL explicitly identifies faulty steps, erases them, and regenerates reasoning in place, preventing defective logic from propagating through the reasoning chain. This targeted correction mechanism turns brittle reasoning into a more resilient process. Models trained with ERL, termed ESearch, achieve substantial improvements on HotpotQA, MuSiQue, 2Wiki, and Bamboogle, with the 3B model achieving +8.48% EM and +11.56% F1, and the 7B model achieving +5.38% EM and +7.22% F1 over previous state-of-the-art(SOTA) results. These findings suggest that erasable reinforcement learning provides a powerful paradigm shift for robust multi-step reasoning in LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
