arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2606.01993cs.CLcs.AIcs.LG

MMG2Skill: 智能体能否从野外指南中提炼出自我进化的技能?

MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills?

Nanjing University(南京大学);Kuaishou Technology(快手科技)

Xinyu Che, Junqi Xiong, Yunfei Ge, Xinping Lei, Shihao Li, Hang Yan, Han Li, Yuanxing Zhang, Zhiqi Bai, Jinhua Hao, Ming Sun, Han Li, Jiaheng Liu

AI总结:

提出MMG2Skill框架,将多模态异构的野外指南编译为可编辑技能,通过轨迹级根因反馈持续改进,在GUI控制、开放游戏和策略卡牌任务中显著提升VLM智能体性能。

摘要:

网络上丰富的程序性知识对于帮助智能体解决长程任务具有巨大潜力。然而,这些知识通常是多模态、异构、有噪声的,并且隐含地假设人类执行者,使得它们难以直接用作智能体所需的技能。为了弥合人类导向指南与智能体可执行技能之间的差距,我们将此问题形式化为指南到技能学习:将野外指南转换为可执行技能,并从智能体可观察的轨迹中持续改进它们。为了评估现有智能体在此任务上的能力,我们引入了MMG2Skill-Bench,这是针对该问题的首个基准测试。我们进一步提出了MMG2Skill,一个闭环框架,它将指南编译为可编辑技能,在执行过程中将固定的视觉语言模型(VLM)智能体条件化于这些技能,并从轨迹级根因反馈中修正技能,而不使用基准测试分数。在GUI控制、开放式游戏和策略卡牌游戏中,使用六个VLM骨干网络,MMG2Skill在每个模型-领域设置中始终优于普通基线智能体,在骨干网络上实现了宏观平均增益+12.8到+25.3个百分点。消融研究表明,直接用原始指南提示智能体会降低性能,而结构化技能构建和轨迹驱动修订对于观察到的改进都是必要的。在成功可推断的任务中,当成功信号适当校准时,基于分析器的提前停止进一步防止了后期性能退化,并节省了25%-53%的尝试次数。

英文摘要:

Abundant procedural knowledge on the Web holds great potential for helping agents solve long-horizon tasks. However, such knowledge is often multimodal, heterogeneous, noisy, and implicitly assumes human executors, making it difficult to use directly as the skills required by agents. To bridge the gap between human-oriented guides and agent-executable skills, we formalize this problem as guide-to-skill learning: converting in-the-wild guides into executable skills and continuously improving them from trajectories observable to the agent. To evaluate the capability of existing agents on this task, we introduce MMG2Skill-Bench, the first benchmark designed for this problem. We further propose MMG2Skill, a closed-loop framework that compiles guides into editable skills, conditions a fixed vision-language model (VLM) agent on these skills during execution, and revises the skills from trajectory-level root-cause feedback without using benchmark scores. Across GUI control, open-ended gameplay, and strategic card play with six VLM backbones, MMG2Skill consistently outperforms vanilla baseline agents in every model-domain setting, achieving macro-average gains of +12.8 to +25.3 percentage points across backbones. Ablation studies show that directly prompting agents with raw guides can degrade performance, while both structured skill construction and trajectory-driven revision are necessary for the observed improvements. On success-inferable tasks, analyzer-based early stopping further prevents late-stage performance regressions and saves 25%-53% of attempts when the success signal is properly calibrated.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑