arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.15915cs.CV
收藏

VIEWCRAFT:演化多模态智能体所见内容及其读取方式

VIEWCRAFT: Evolving What Multimodal Agents See and How They Read It

PKU(北京大学);MIT-IBM Computing Research Lab(麻省理工学院-IBM计算研究实验室);UW–Madison(威斯康星大学麦迪逊分校);CASIA(中国科学院自动化研究所);CMU(卡内基梅隆大学);PolyU(香港理工大学);HKU(香港大学)

Binxiao Xu, Ruichuan An, Bocheng Zou, Weihua Du, Xinyu Wei, Hangyu Guo, YiFan Zhang, Renye Yan, Chengqi Duan, Hang Hua

AI总结:

ViewCraft协同演化多模态智能体的证据接口与读取过程,在8个基准上平均提升任务分数6.1个百分点,学习到的技能可跨执行器迁移且性能优于直接执行。

摘要:

可复用技能无需更新模型权重即可提升多模态智能体的性能,但其有效性既取决于所呈现的证据,也取决于读取该证据的过程。我们提出ViewCraft,它可协同演化可执行的证据接口及其读取过程。该接口负责选择、转换和组织观测结果;读取器则解释生成的视图。失败轨迹指导修订,重放与配对评估保留有用的变更,而接口投影则从符合条件的被拒绝联合提案中恢复视觉编辑。在8个基准测试中,ViewCraft相比直接执行的任务分数平均提升6.1个百分点,且在所有基准上均有增益。在匹配的输入令牌预算下,该优势依然存在。在全部5项多源任务中,学习到的接口比初始技能少使用8%-15%的输入令牌。这些学习到的技能还可跨执行器迁移,在所有评估的迁移场景中均优于直接执行,证明证据构建与读取是一项耦合的可复用技能。

英文摘要:

Reusable skills improve multimodal agents without updating model weights, but their effectiveness depends on both the evidence presented and the procedure used to read it. We introduce ViewCraft, which jointly evolves an executable evidence interface and its reading procedure. The interface selects, transforms, and organizes observations; the reader interprets the resulting views. Failure traces guide revisions, replay and paired evaluation retain useful changes, and interface projection recovers visual edits from eligible rejected joint proposals. Across eight benchmarks, ViewCraft improves over direct execution by an average of $6.1$ points in task score, with gains on every benchmark. The advantage persists at a matched input-token budget. On all five multi-source tasks, the learned interface uses 8%-15% fewer input tokens than Initial skill. The learned skills also transfer across executors and outperform direct execution in every evaluated transfer, supporting evidence construction and reading as a coupled, reusable skill.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑