SegAgent：通过模仿人类标注者轨迹探索多模态大语言模型的像素理解能力
SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories
AI总结：
本文提出类人掩码标注任务HLMAT，将分割建模为多步马尔可夫决策过程，使MLLM通过文本点击点迭代生成掩码，并据此微调得到SegAgent模型，性能媲美SOTA且支持掩码细化等任务。
摘要：
尽管多模态大语言模型（MLLMs）已展现出足够的图像理解能力，但它们在像素级理解方面仍存在困难，这限制了其实际应用。现有的评估任务如视觉问答（VQA）和视觉定位仍然过于粗糙，难以准确评估细粒度像素理解能力。虽然分割是像素级理解的基础，但现有方法通常要求 MLLMs 生成隐式 token，并通过外部像素解码器进行解码。这种方法破坏了 MLLM 的文本输出空间，可能损害语言能力并降低灵活性和可扩展性，同时无法反映模型内在的像素级理解能力。因此，我们提出了类人掩码标注任务（HLMAT），这是一种新范式，让 MLLMs 使用交互式分割工具模仿人类标注者。HLMAT 将分割建模为多步马尔可夫决策过程（Markov Decision Process），使 MLLMs 能够迭代地生成基于文本的点击点，无需架构改动或隐式 token 即可获得高质量掩码。基于这一设置，我们开发了 SegAgent，一个在类人标注轨迹上微调的模型，其性能可与最先进（SOTA）方法相媲美，并支持掩码细化和标注过滤等额外任务。HLMAT 为评估 MLLMs 的细粒度像素理解提供了一套协议，并引入了一个以视觉为中心的多步决策任务，有助于探索 MLLMs 的视觉推理能力。我们对策略改进方法 StaR 和 PRM 引导的树搜索的适配进一步增强了模型在复杂分割任务中的鲁棒性，为未来 MLLMs 在细粒度视觉感知和多步决策方面的进展奠定了基础。
英文摘要：
While MLLMs have demonstrated adequate image understanding capabilities, they still struggle with pixel-level comprehension, limiting their practical applications. Current evaluation tasks like VQA and visual grounding remain too coarse to assess fine-grained pixel comprehension accurately. Though segmentation is foundational for pixel-level understanding, existing methods often require MLLMs to generate implicit tokens, decoded through external pixel decoders. This approach disrupts the MLLM's text output space, potentially compromising language capabilities and reducing flexibility and extensibility, while failing to reflect the model's intrinsic pixel-level understanding. Thus, we introduce the Human-Like Mask Annotation Task (HLMAT), a new paradigm where MLLMs mimic human annotators using interactive segmentation tools. Modeling segmentation as a multi-step Markov Decision Process, HLMAT enables MLLMs to iteratively generate text-based click points, achieving high-quality masks without architectural changes or implicit tokens. Through this setup, we develop SegAgent, a model fine-tuned on human-like annotation trajectories, which achieves performance comparable to state-of-the-art (SOTA) methods and supports additional tasks like mask refinement and annotation filtering. HLMAT provides a protocol for assessing fine-grained pixel understanding in MLLMs and introduces a vision-centric, multi-step decision-making task that facilitates exploration of MLLMs' visual reasoning abilities. Our adaptations of policy improvement method StaR and PRM-guided tree search further enhance model robustness in complex segmentation tasks, laying a foundation for future advancements in fine-grained visual perception and multi-step decision-making for MLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
