arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.28201cs.AI
收藏

MaD-RSI:用于智能体控制策略递归自我改进的模型即数据

MaD-RSI: Model-as-Data for Recursive Self-Improvement of Agent Harnesses

Beijing Institute of Technology(北京理工大学);Tsinghua University(清华大学);Chengdu Institute of Computer Applications, Chinese Academy of Sciences(中国科学院成都计算机应用研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing University of Technology(北京工业大学);Yangtze Delta Region Institute of Tsinghua University, Zhejiang(浙江清华长三角研究院)

Yu Cui, Hong He, Ruiqing Yue, Sicheng Pan, Zhuoyu Sun, Yifei Liu, Ting Li, Wenzhuo Zhu, Baohan Huang, Haibin Zhang, Cong Zuo

AI总结:

本研究提出MaD-RSI框架,将模型视为数据,利用多个异构LLM的失败轨迹优化共享控制策略,提升了控制策略跨模型的泛化性并降低了训练成本。

摘要:

智能体控制策略(agent harness)的递归自我改进(RSI)已成为增强大语言模型(LLM)智能体能力的有效方法。通过在给定数据集上针对特定LLM搜索控制策略空间,RSI可发现适配该模型能力与行为特征的控制策略。然而,此类优化常导致学习到的控制策略与其目标LLM间产生强耦合,限制了跨模型的泛化能力,且适配新LLM需大量计算资源。此外,现有RSI方法通常仅利用单个模型的反馈优化控制策略,且仅在训练后评估其对其他模型的泛化性。该范式将模型差异主要视为评估泛化性的标准,而非优化的有价值证据。本文提出新视角,将模型视为数据,把控制策略置于优化核心。基于此视角,我们提出MaD-RSI框架,该框架通过将多个异构LLM的不同失败轨迹纳入训练过程,直接优化跨这些模型的共享控制策略。MaD-RSI并非仅在优化后评估模型差异,而是将这些差异作为训练证据,指导共享控制策略的自我改进。此外,我们引入含两个互补指标的新评估框架,用于评估控制策略跨模型优化的有效性。在多个数据集与LLM上开展的实验表明,MaD-RSI较现有方法提升了泛化性,同时降低了训练成本。

英文摘要:

Recursive self-improvement (RSI) of agent harnesses has emerged as an effective approach to enhancing the capabilities of large language model (LLM) agents. By searching the harness space for a specific LLM on a given dataset, RSI can discover a harness tailored to the capabilities and behavioral characteristics of that model. However, such optimization often creates a strong coupling between the learned harness and its target LLM, limiting generalization across models and requiring substantial computational resources to adapt to new LLMs. Moreover, existing RSI methods typically optimize a harness using feedback from a single model and evaluate its generalization to other models only after training. This paradigm treats model differences primarily as a criterion for evaluating generalization rather than as valuable evidence for optimization. In this paper, we introduce a new perspective that treats models as data and places the harness at the center of optimization. Building on this perspective, we propose \textsc{MaD-RSI}, a framework that directly optimizes a shared harness across multiple heterogeneous LLMs by incorporating their diverse failure trajectories into the training process. Rather than assessing model differences only after optimization, \textsc{MaD-RSI} leverages these differences as training evidence to guide the self-improvement of a shared harness. Furthermore, we introduce a new evaluation framework with two complementary metrics designed to assess the effectiveness of harness optimization across models. Experiments across multiple datasets and LLMs demonstrate that \textsc{MaD-RSI} improves generalization over existing approaches while reducing training costs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑