PixelRefer：支持任意粒度时空目标指代的统一框架
PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularity
AI总结：
提出统一的区域级多模态框架PixelRefer及其轻量变体PixelRefer-Lite，通过尺度自适应目标分词器与全局上下文预融合模块，结合高质量指令数据集，在图像和视频中实现高效细粒度的目标级推理。
摘要：
多模态大语言模型（MLLMs）在开放世界视觉理解中展现了强大的通用能力。然而，现有大多数MLLMs主要侧重于整体、场景级的理解，往往忽略了细粒度、以目标为中心的推理需求。本文提出PixelRefer，一个统一的区域级MLLM框架，能在图像和视频中实现对用户指定区域的高级细粒度理解。基于LLM注意力主要集中于目标级token的观察，我们提出尺度自适应目标分词器（SAOT），从自由形状区域生成紧凑且语义丰富的目标表示。分析表明全局视觉token主要在LLM早期层起作用，这启发了PixelRefer-Lite的设计，该高效变体采用以目标为中心的注入模块，将全局上下文预融合到目标token中。这产生了一个轻量级的纯目标框架，在保持高语义保真度的同时大幅降低了计算成本。为促进细粒度指令微调，我们构建了PixelRefer-2.2M，一个高质量的以目标为中心的指令数据集。跨多个基准的广泛实验验证了PixelRefer以更少训练样本实现领先性能，而PixelRefer-Lite提供了具竞争力的准确率并在效率上取得显著提升。
英文摘要：
Multimodal large language models (MLLMs) have demonstrated strong general-purpose capabilities in open-world visual comprehension. However, most existing MLLMs primarily focus on holistic, scene-level understanding, often overlooking the need for fine-grained, object-centric reasoning. In this paper, we present PixelRefer, a unified region-level MLLM framework that enables advanced fine-grained understanding over user-specified regions across both images and videos. Motivated by the observation that LLM attention predominantly focuses on object-level tokens, we propose a Scale-Adaptive Object Tokenizer (SAOT) to generate compact and semantically rich object representations from free-form regions. Our analysis reveals that global visual tokens contribute mainly in early LLM layers, inspiring the design of PixelRefer-Lite, an efficient variant that employs an Object-Centric Infusion module to pre-fuse global context into object tokens. This yields a lightweight Object-Only Framework that substantially reduces computational cost while maintaining high semantic fidelity. To facilitate fine-grained instruction tuning, we curate PixelRefer-2.2M, a high-quality object-centric instruction dataset. Extensive experiments across a range of benchmarks validate that PixelRefer achieves leading performance with fewer training samples, while PixelRefer-Lite offers competitive accuracy with notable gains in efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
