更优、更强、更快：在基于多模态大语言模型的分割中解决三重困境
Better, Stronger, Faster: Tackling the Trilemma in MLLM-based Segmentation with Simultaneous Textual Mask Prediction
AI总结：
STAMP通过同时预测文本和分割掩码，解决了多模态大语言模型在保持对话能力、高分割性能和快速推理间的三重困境。
摘要：
将分割整合到多模态大语言模型（MLLMs）中提出了一个核心三重困境：同时保持对话能力、实现高分割性能和确保快速推理。现有方法被迫做出妥协。嵌入预测方法引入了一个冲突的像素级目标，会降低MLLM的通用对话能力。替代方案是下一个令牌预测，将分割重新定义为自回归任务，这虽然保留了对话能力，但迫使在稀疏输出时分割性能较差或在丰富输出时推理速度变得极慢。我们通过所有掩码预测这一新范式解决这一三重困境，该范式将自回归对话生成与非自回归掩码预测解耦。我们提出了STAMP：同时文本所有掩码预测，一种体现这一范式的MLLM。在生成文本响应后，STAMP通过将分割掩码视为图像块上的并行“填空”任务，在单次前向传递中预测整个分割掩码。这种设计通过避免冲突目标保持了MLLM的对话能力，通过利用丰富的双向空间上下文为所有掩码标记实现了高分割性能，并实现了极高的速度。广泛的实验表明，STAMP在多个分割基准测试中显著优于最先进的方法，提供了一个在对话、分割和速度上均无妥协的解决方案。
英文摘要：
Integrating segmentation into Multimodal Large Language Models (MLLMs) presents a core trilemma: simultaneously preserving dialogue ability, achieving high segmentation performance, and ensuring fast inference. Prevailing paradigms are forced into a compromise. Embedding prediction methods introduce a conflicting pixel-level objective that degrades the MLLM's general dialogue abilities. The alternative, next-token prediction, reframes segmentation as an autoregressive task, which preserves dialogue but forces a trade-off between poor segmentation performance with sparse outputs or prohibitive inference speeds with rich ones. We resolve this trilemma with all-mask prediction, a novel paradigm that decouples autoregressive dialogue generation from non-autoregressive mask prediction. We present STAMP: Simultaneous Textual All-Mask Prediction, an MLLM that embodies this paradigm. After generating a textual response, STAMP predicts an entire segmentation mask in a single forward pass by treating it as a parallel "fill-in-the-blank" task over image patches. This design maintains the MLLM's dialogue ability by avoiding conflicting objectives, enables high segmentation performance by leveraging rich, bidirectional spatial context for all mask tokens, and achieves exceptional speed. Extensive experiments show that STAMP significantly outperforms state-of-the-art methods across multiple segmentation benchmarks, providing a solution that excels in dialogue, segmentation, and speed without compromise.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
