重新思考持续经验内化以实现自我进化的大语言模型智能体
Rethinking Continual Experience Internalization for Self-Evolving LLM Agents
AI总结：
本文通过经验粒度、注入模式和内化机制三个维度，提出一种稳定可持续的经验内化方法，解决多轮经验学习中的能力崩溃问题。
摘要：
经验内化将过去交互中的上下文经验转化为可重用的参数化能力，为大型语言模型（LLM）的持续学习提供了一条有前景的路径。虽然先前的工作主要关注单次迭代迁移，但我们发现在多轮经验学习下，现有方法遭受的是渐进的能力崩溃而非复合改进。我们通过经验内化的三个关键维度系统地考察了这种失败：（1）经验粒度：我们发现原则级经验比实例级经验更持久，因为它有效地从轨迹特定细节中抽象出可迁移的策略。（2）经验注入模式：我们的分析表明，逐步注入通过将经验与中间决策状态对齐，显著优于全局注入，这一特性对于长程工具使用至关重要。（3）内化机制：我们证明，在高质量教师轨迹上的离策略上下文蒸馏提供了比在策略上下文蒸馏更稳定的训练信号，后者固有地受限于对学生诱导的缺陷状态的局部修正。这些见解共同产生了一个简单而稳健的配方，用于稳定和可持续的经验内化，为工程化自我进化和持续学习的LLM提供了具体指导。
英文摘要：
Experience internalization converts contextual experience from past interactions into reusable parametric capability, offering a promising path toward continual learning in large language models (LLMs). While prior work has predominantly focused on single-iteration transfer, we discover that under multi-iteration experience learning, existing methods suffer from a progressive capability collapse rather than compounding improvement. We systematically examine this failure through three vital dimensions of experience internalization: (1) Experience Granularity: We find that principle-level experience is more durable than instance-level experience, as it effectively abstracts transferable strategies away from trajectory-specific details. (2) Experience Injection Pattern: Our analysis reveals that step-wise injection significantly outperforms global injection by aligning experience with intermediate decision states, a property that is critical for long-horizon tool use. (3) Internalization Regime: We demonstrate that off-policy context-distillation on high-quality teacher trajectories provides a substantially more stable training signal than on-policy context-distillation, which is inherently limited by local corrections on student-induced flawed states. Together, these insights yield a simple yet robust recipe for stable and sustainable experience internalization, providing concrete guidance for engineering self-evolving and continually learning LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
