推理以关注：尝试理解<SEG>标记的作用
Reasoning to Attend: Try to Understand How <SEG> Token Works
AI总结：
本文研究了<SEG>标记在多模态模型中的作用，通过可视化相似性图揭示其在图像-文本对中的语义相似性贡献，并提出READ方法提升模型的推理能力。
摘要：
当前大型多模态模型（LMMs）通过<SEG>标记作为文本提示联合优化视觉语言模型（如LLaVA）和下游任务特定模型（如SAM）。然而，我们发现很少有研究探讨其工作机制。本文首先可视化相似性图，通过计算<SEG>标记与图像标记嵌入的语义相似性获得。有趣的是，相似性图中存在显著的一致性，揭示了<SEG>标记在图像-文本对中的语义相似性贡献。具体而言，<SEG>标记作为文本词汇中的占位符，广泛查询单个图像块以匹配文本中对象的语义，同时LLMs在微调过程中。基于上述发现，我们提出READ，通过高激活点指导的相似性图促进LMMs的鲁棒推理能力，即在指导下实现对注意力的合理分配。READ包含直观的Similarity as Points模块（SasP），可无缝应用于<SEG>类范式。此外，在ReasonSeg和RefCOCO(+/g)数据集上进行了大量实验，并在增强的FP-RefCOCO(+/g)数据集上验证了READ在微调后的生成能力。所有代码和模型均在https://github.com/rui-qian/READ上公开。
英文摘要：
Current Large Multimodal Models (LMMs) empowered visual grounding typically rely on $\texttt{<SEG>}$ tokens as a text prompt to jointly optimize the vision-language model (e.g., LLaVA) and the downstream task-specific model (e.g., SAM). However, we observe that little research has looked into how it works.In this work, we first visualize the similarity maps, which are obtained by computing the semantic similarity between the $\texttt{<SEG>}$ token and the image token embeddings derived from the last hidden layer in both the LLaVA encoder and SAM decoder. Intriguingly, we have found that a striking consistency holds in terms of activation responses in the similarity map, which reveals that what the $\texttt{<SEG>}$ token contributes to is semantic similarity within image-text pairs. Specifically, the $\texttt{<SEG>}$ token, a placeholder expanded in text vocabulary, extensively queries among individual tokenized image patches to match the semantics of an object from text to the paired image, while the Large Language Models (LLMs) are being fine-tuned. Upon the above findings, we present READ, which facilitates LMMs' resilient $\textbf{REA}$soning capability of where to atten$\textbf{D}$ under the guidance of highly activated points borrowed from similarity maps. Remarkably, READ features an intuitive design, Similarity as Points module (SasP), which can be seamlessly applied to $\texttt{<SEG>}$-like paradigms in a plug-and-play fashion. Also, extensive experiments have been conducted on ReasonSeg and RefCOCO(+/g) datasets. To validate whether READ suffers from catastrophic forgetting of previous skills after fine-tuning, we further assess its generation ability on an augmented FP-RefCOCO(+/g) dataset. All codes and models are publicly available at https://github.com/rui-qian/READ.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
