arXiv 2503.130262025-07-16cs.CV

HiMTok：学习用于大型多模态模型图像分割的层次化掩码Token

HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Model

Uni-Ubi；Zhejiang University（浙江大学）；Tongji University（同济大学）

Tao Wang, Changxu Cheng, Lingfeng Wang, Senda Chen, Wuyue Zhao

备注：: Accepted by ICCV 2025; the code is at https://github.com/yayafengzi/LMM-HiMTok

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出层次化掩码分词器HiMTok，用最多32个Token表示分割掩码，免除了去分词化对原始图像的依赖，通过3阶段训练方案实现由粗到细学习与多任务双向信息流，在多项分割任务上达到SOTA并增强视觉定位与理解能力。

摘要：

大型多模态模型（LMMs）的卓越性能引起了图像分割社区的极大兴趣。为了与下一Token预测范式对齐，当前LMM驱动的分割方法要么使用对象边界点来表示掩码，要么引入特殊的分割Token，其隐藏状态由需要原始图像作为输入的分割模型解码。然而，这些方法通常面临掩码表示不充分和架构复杂的问题，限制了LMM的潜力。在本研究中，我们提出了层次化掩码分词器（HiMTok），它用最多32个Token表示分割掩码，并消除了掩码去分词化过程中对原始图像的需求。HiMTok允许紧凑且由粗到细的掩码表示，与LLM下一Token预测范式良好对齐，并促进直接获取分割能力。我们开发了一个3阶段训练方案，用于渐进式学习分割和视觉能力，其特点是通过层次化掩码损失实现有效的由粗到细学习。此外，我们启用了双向信息流，允许边界框和掩码Token之间的转换，以充分利用多任务训练潜力。大量实验表明，我们的方法在各种分割任务中实现了最先进的性能，同时增强了视觉定位并保持了整体视觉理解能力。

英文摘要：

The remarkable performance of large multimodal models (LMMs) has attracted significant interest from the image segmentation community. To align with the next-token-prediction paradigm, current LMM-driven segmentation methods either use object boundary points to represent masks or introduce special segmentation tokens, whose hidden states are decoded by a segmentation model requiring the original image as input. However, these approaches often suffer from inadequate mask representation and complex architectures, limiting the potential of LMMs. In this work, we propose the Hierarchical Mask Tokenizer (HiMTok), which represents segmentation masks with up to 32 tokens and eliminates the need for the original image during mask de-tokenization. HiMTok allows for compact and coarse-to-fine mask representations, aligning well with the LLM next-token-prediction paradigm and facilitating the direct acquisition of segmentation capabilities. We develop a 3-stage training recipe for progressive learning of segmentation and visual capabilities, featuring a hierarchical mask loss for effective coarse-to-fine learning. Additionally, we enable bidirectional information flow, allowing conversion between bounding boxes and mask tokens to fully leverage multi-task training potential. Extensive experiments demonstrate that our method achieves state-of-the-art performance across various segmentation tasks,while also enhancing visual grounding and maintaining overall visual understanding.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。