arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.04560cs.AI
收藏

ContextNav:迈向智能体多模态上下文学习

ContextNav: Towards Agentic Multimodal In-Context Learning

The University of Queensland(昆士兰大学);Nanjing University(南京大学);University of California, Los Angeles(加州大学洛杉矶分校);University of California, Merced(加州大学默塞德分校)

Honghao Fu, Yuan Ouyang, Kai-Wei Chang, Yiwei Wang, Zi Huang, Yujun Cai

AI总结:

ContextNav提出首个智能体框架,结合自动检索与类人策展,通过图编排实现噪声鲁棒且动态优化的多模态上下文学习,在多个数据集上取得最先进性能。

摘要:

近期进展表明,多模态大语言模型(MLLMs)展现出强大的多模态上下文学习(ICL)能力,使其能够从少量上下文示例中适应新的视觉-语言任务。然而,现有的ICL方法在兼顾可扩展性与鲁棒性方面面临挑战,尤其是在多样任务和含噪上下文示例的场景下:手动选择示例可产生干净的上下文,但劳动密集且针对特定任务;而基于相似性的检索虽提升了可扩展性,却可能引入不相关或结构不一致的样本,从而降低ICL性能。为解决这些局限,我们提出ContextNav,这是首个将自动检索的可扩展性与类人策展的质量和适应性相结合的智能体框架,为多模态ICL实现噪声鲁棒且动态优化的上下文构建。ContextNav在由基于图的编排驱动的闭环工作流中统一了上下文管理和噪声鲁棒上下文构建。具体而言,它构建了一个资源感知的多模态嵌入流水线,维护一个可检索的向量数据库,并应用智能体检索和结构对齐来构造抗噪上下文。操作语法图(OGG)进一步支持自适应工作流规划和优化,使智能体能够根据下游ICL反馈改进其操作策略。实验结果表明,ContextNav在多个数据集上取得了最先进的性能,凸显了智能体工作流在推进多模态ICL中可扩展且鲁棒的上下文构建方面的潜力。

英文摘要:

Recent advances demonstrate that multimodal large language models (MLLMs) exhibit strong multimodal in-context learning (ICL) capabilities, enabling them to adapt to novel vision-language tasks from a few contextual examples. However, existing ICL approaches face challenges in reconciling scalability with robustness across diverse tasks and noisy contextual examples: manually selecting examples produces clean contexts but is labor-intensive and task-specific, while similarity-based retrieval improves scalability but could introduce irrelevant or structurally inconsistent samples that degrade ICL performance. To address these limitations, we propose ContextNav, the first agentic framework that integrates the scalability of automated retrieval with the quality and adaptiveness of human-like curation, enabling noise-robust and dynamically optimized contextualization for multimodal ICL. ContextNav unifies context management and noise-robust contextualization within a closed-loop workflow driven by graph-based orchestration. Specifically, it builds a resource-aware multimodal embedding pipeline, maintains a retrievable vector database, and applies agentic retrieval and structural alignment to construct noise-resilient contexts. An Operational Grammar Graph (OGG) further supports adaptive workflow planning and optimization, enabling the agent to refine its operational strategies based on downstream ICL feedback. Experimental results demonstrate that ContextNav achieves state-of-the-art performance across various datasets, underscoring the promise of agentic workflows for advancing scalable and robust contextualization in multimodal ICL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑