面向多模态引导视频目标分割的智能体AI
Towards Agentic AI for Multimodal-Guided Video Object Segmentation
AI总结：
针对基于指代的视频目标分割中固定流程缺乏灵活性的问题，提出Multi-Modal Agent智能体系统，利用LLM生成动态工作流并迭代调用多模态专用工具，在RVOS和Ref-AVS任务上取得显著性能提升。
摘要：
基于指代的视频目标分割是一个多模态问题，需要在外部线索的引导下生成细粒度的分割结果。该任务的传统方法通常涉及训练专门化模型，这带来了高计算复杂度和大量人工标注成本。视觉-语言基础模型的最新进展为免训练方法开辟了充满前景的方向。多项研究探索了利用这些通用模型进行细粒度分割，取得了与全监督的任务特定模型相当的性能。然而，现有方法依赖固定流程，缺乏适应任务动态特性所需的灵活性。为了解决这一局限性，我们提出了Multi-Modal Agent，这是一种新颖的智能体系统，旨在以更灵活、更具适应性的方式解决该任务。具体而言，我们的方法利用大型语言模型（LLMs）的推理能力，生成针对每个输入量身定制的动态工作流。这一自适应过程与一组专为跨不同模态的低层任务设计的专用工具进行迭代交互，以识别由多模态线索描述的目标对象。我们的智能体方法在两项多模态条件下的VOS任务（RVOS和Ref-AVS）上，展现出相较于先前方法的显著改进。
英文摘要：
Referring-based Video Object Segmentation is a multimodal problem that requires producing fine-grained segmentation results guided by external cues. Traditional approaches to this task typically involve training specialized models, which come with high computational complexity and manual annotation effort. Recent advances in vision-language foundation models open a promising direction toward training-free approaches. Several studies have explored leveraging these general-purpose models for fine-grained segmentation, achieving performance comparable to that of fully supervised, task-specific models. However, existing methods rely on fixed pipelines that lack the flexibility needed to adapt to the dynamic nature of the task. To address this limitation, we propose Multi-Modal Agent, a novel agentic system designed to solve this task in a more flexible and adaptive manner. Specifically, our method leverages the reasoning capabilities of large language models (LLMs) to generate dynamic workflows tailored to each input. This adaptive procedure iteratively interacts with a set of specialized tools designed for low-level tasks across different modalities to identify the target object described by the multimodal cues. Our agentic approach demonstrates clear improvements over prior methods on two multimodal-conditioned VOS tasks: RVOS and Ref-AVS.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
