IBISAgent: 通过MLLMs增强像素级视觉推理以实现通用生物医学对象指称与分割
IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation
AI总结：
本文提出IBISAgent，通过多步决策过程增强MLLMs的像素级视觉推理能力，解决现有分割方法在隐式分割标记和迭代优化方面的不足，提升生物医学分割任务的性能。
摘要：
近年来，医疗MLLMs的研究逐渐从图像级理解转向细粒度的像素级理解。尽管分割是像素级理解的基础，但现有方法面临两大挑战：首先，它们引入隐式分割标记，并需要同时微调MLLM和外部像素解码器，增加了灾难性遗忘的风险并限制了对非领域场景的泛化能力。其次，大多数方法依赖单次推理，缺乏迭代优化分割结果的能力，导致性能不佳。为克服这些限制，我们提出了一种新的代理MLLM，称为IBISAgent，将分割重新表述为以视觉为中心的多步骤决策过程。IBISAgent使MLLMs能够生成交错的推理和基于文本的点击操作，调用分割工具，并在不修改架构的情况下生成高质量的掩码。通过在掩码图像特征上进行多步骤视觉推理，IBISAgent自然支持掩码细化，并促进像素级视觉推理能力的发展。我们进一步设计了一个两阶段训练框架，包括冷启动监督微调和具有定制精细奖励的代理强化学习，增强模型在复杂医学指称和推理分割任务中的鲁棒性。广泛的实验表明，IBISAgent在封闭源和开源SOTA方法上均表现优异。
英文摘要：
Recent research on medical MLLMs has gradually shifted its focus from image-level understanding to fine-grained, pixel-level comprehension. Although segmentation serves as the foundation for pixel-level understanding, existing approaches face two major challenges. First, they introduce implicit segmentation tokens and require simultaneous fine-tuning of both the MLLM and external pixel decoders, which increases the risk of catastrophic forgetting and limits generalization to out-of-domain scenarios. Second, most methods rely on single-pass reasoning and lack the capability to iteratively refine segmentation results, leading to suboptimal performance. To overcome these limitations, we propose a novel agentic MLLM, named IBISAgent, that reformulates segmentation as a vision-centric, multi-step decision-making process. IBISAgent enables MLLMs to generate interleaved reasoning and text-based click actions, invoke segmentation tools, and produce high-quality masks without architectural modifications. By iteratively performing multi-step visual reasoning on masked image features, IBISAgent naturally supports mask refinement and promotes the development of pixel-level visual reasoning capabilities. We further design a two-stage training framework consisting of cold-start supervised fine-tuning and agentic reinforcement learning with tailored, fine-grained rewards, enhancing the model's robustness in complex medical referring and reasoning segmentation tasks. Extensive experiments demonstrate that IBISAgent consistently outperforms both closed-source and open-source SOTA methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
