arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.19475cs.CVcs.CL
收藏

ActICL:面向智能体多模态上下文学习的后训练方法

ActICL: Post-Training for Agentic Multimodal In-Context Learning

University of Texas at Austin(德克萨斯大学奥斯汀分校);Rutgers University(罗格斯大学)

Yanshu Li, Ruixiang Tang, Junyi Jessy Li, Elias Stengel-Eskin

AI总结:

本文提出ActICL,通过工具增强推理训练MLLM智能体优化多模态ICL的演示检索,在7个基准和3种MLLM上性能优于现有方法,其获取的上下文还可提升前沿模型的ICL表现。

摘要:

多模态上下文学习(ICL)是多模态大语言模型(MLLM)的重要能力,其性能与所用演示示例密切相关,因此演示检索是ICL的核心问题。现有常用方法采用固定评分方法一次性检索所有演示,可能引入冗余和噪声;上下文感知检索方法虽通过允许从不断增长的上下文进行增量检索来解决该问题,但均未让MLLM学会自行判断所需新演示并进行渐进式检索。为此,本文提出ActICL,通过工具增强推理训练MLLM智能体以获取和使用ICL上下文。ActICL智能体在迭代推理和工具使用过程中分析自身信息缺口,基于缺口执行针对性检索获取初始候选集,再利用现有上下文进一步筛选候选集并优化上下文,或迭代执行新的检索;当智能体判断上下文足以完成任务,或达到资源限制且满足用户请求时,将获取的上下文发送至回答模型(可为智能体自身或其他MLLM)。本文采用两阶段后训练流程,联合优化智能体的上下文获取与回答过程。在7个基准测试和3种MLLM上,ActICL采用自适应数量的演示示例,相比基于相似度的标准检索方法平均性能提升11.6个百分点,相比最强的现有基线方法提升4.4个百分点;ActICL智能体获取的上下文也可被前沿模型有效利用,当GPT-5.6-Sol执行4次演示(4-shot)ICL时,ActICL的上下文在3个基准测试上的平均性能比标准检索提升3.5个百分点。

英文摘要:

Multimodal in-context learning (ICL) is an important capability of multimodal large language models (MLLMs). Its performance is closely tied to the demonstrations it uses, making demonstration retrieval a core problem for ICL. Common methods use fixed scoring methods to retrieve all the demonstrations at once, which may introduce redundancy and noise. Context-aware retrieval methods have emerged to address this problem by allowing incremental retrieval from a growing context. However, none of them lets the MLLM learn how to judge what new demonstrations it needs and then carry out progressive retrieval on its own. To this end, we introduce ActICL, which trains an MLLM agent to acquire and use an ICL context via tool-augmented reasoning. An ActICL agent analyzes its own information gaps during iterative reasoning and tool-use actions. Based on these gaps, it performs targeted retrieval to acquire a set of initial candidates. It can then use the existing context to further filter the candidates and refine the context, or it can iteratively perform a new retrieval. When the agent judges that the context is sufficient for the task, or when the resource limit is reached with the user request met, it sends the acquired context to an answering model, either the agent itself or another MLLM. We use a two-stage post-training procedure that jointly optimizes the agent's acquisition and answering. With an adaptive number of demonstrations, ActICL improves average performance by 11.6 percentage points over standard similarity-based retrieval across seven benchmarks and three MLLMs, and by 4.4 points compared to the strongest prior baseline. The contexts acquired by ActICL agents can also be used effectively by frontier models. When GPT-5.6-Sol performs 4-shot ICL, ActICL's contexts improve its performance by 3.5 points over standard retrieval, averaged over three benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑