长远时间LLM智能体的回顾信用分配
Hindsight Credit Assignment for Long-Horizon LLM Agents
AI总结：
HCAPO通过整合回顾信用分配提升LLM智能体在长期任务中的探索效率和决策简洁性。
摘要：
大型语言模型（LLM）智能体在长期、多步骤任务中常常面临显著的信用分配挑战，因为奖励稀疏。现有的价值无关方法，如群体相对策略优化（GRPO），遇到两个基本瓶颈：步骤级Q值估计不准确和中间状态的价值基线不一致。为了解决这些限制，我们引入HCAPO，这是第一个将回顾信用分配整合到LLM智能体中的框架。HCAPO利用LLM本身作为事后批评者，通过回顾推理来细化步骤级Q值。此外，HCAPO的多尺度优势机制有效补充了关键决策状态中不准确的价值基线。在三个具有挑战性的基准测试中，包括WebShop和ALFWorld，评估表明HCAPO在性能上始终优于最先进的强化学习方法。值得注意的是，HCAPO在使用Qwen2.5-7B-Instruct模型时，WebShop的成功率提高了7.7%，ALFWorld提高了13.8%。这些结果表明，HCAPO显著提高了探索效率，促进了简洁的决策，并确保了在复杂、长期任务中的可扩展性。
英文摘要：
Large Language Model (LLM) agents often face significant credit assignment challenges in long-horizon, multi-step tasks due to sparse rewards. Existing value-free methods, such as Group Relative Policy Optimization (GRPO), encounter two fundamental bottlenecks: inaccurate step-level Q-value estimation and misaligned value baselines for intermediate states. To address these limitations, we introduce HCAPO, the first framework to integrate hindsight credit assignment into LLM agents. HCAPO leverages the LLM itself as a post-hoc critic to refine step-level Q-values through hindsight reasoning. Furthermore, HCAPO's multi-scale advantage mechanism effectively supplements the inaccurate value baselines at critical decision states. Evaluations across three challenging benchmarks, including WebShop and ALFWorld, demonstrate that HCAPO consistently outperforms state-of-the-art RL methods. Notably, HCAPO achieves a 7.7% improvement in success rate on WebShop and a 13.8% on ALFWorld over GRPO using the Qwen2.5-7B-Instruct model. These results indicate that HCAPO significantly enhances exploration efficiency, promotes concise decision-making, and ensures scalability in complex, long-horizon tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
