arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.07413cs.CV
收藏

基于三元组的引用范式用于统一视觉解码

REF-VLM: Triplet-Based Referring Paradigm for Unified Visual Decoding

School of Computer Science, Shanghai Jiao Tong University, Shanghai, 200240, China(上海交通大学计算机科学学院);Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo, China(宁波数字孪生研究院);School of Information Science and Electronic Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China(上海交通大学信息科学与电子工程学院)

Yan Tai, Luhao Zhu, Yunan Ding, Yiying Dong, Guangtao Zhai, Xiaohong Liu, Guodong Guo

AI总结:

REF-VLM通过引入基于三元组的引用范式,提升多任务视觉解码的性能和适应性。

摘要:

多模态大语言模型(MLLMs)在训练于超大规模数据集后,能够展现出在多种视觉-语言任务中强大的零样本能力。然而,对于密集预测任务,如语义分割和关键点检测,MLLMs在仅以文本输出表示时面临显著挑战。同时,当前利用潜在嵌入进行视觉任务解码的MLLMs在多任务学习和多粒度场景中表现出有限的适应性。在本工作中,我们提出了REF-VLM,一个用于统一训练各种视觉解码任务的端到端框架。为应对复杂的视觉解码场景,我们引入了基于三元组的引用范式(TRP),通过三元组结构显式解耦视觉解码任务中的三个关键维度:概念、解码类型和目标。TRP利用符号分隔符来强制结构化表示学习,提高模型输出的可解析性和可解释性。此外,我们构建了视觉-任务指令遵循数据集(VT-Instruct),这是一个包含超过1亿个跨25种任务类型的多任务数据集。除了文本输入和输出外,VT-Instruct还包含各种视觉提示,如点、框、涂鸦和掩码,并生成由文本和视觉单元(如框、关键点、深度和掩码)组成的输出。不同视觉提示和视觉单元的组合生成了多种任务类型,显著扩展了REF-VLM的应用范围。定性和定量实验表明,我们的REF-VLM在多种标准基准上均优于其他MLLMs。代码、数据集和演示将向公众开放。

英文摘要:

Multimodal Large Language Models (MLLMs) demonstrate robust zero-shot capabilities across diverse vision-language tasks after training on mega-scale datasets. However, dense prediction tasks, such as semantic segmentation and keypoint detection, pose significant challenges for MLLMs when represented solely as text outputs. Simultaneously, current MLLMs utilizing latent embeddings for visual task decoding generally demonstrate limited adaptability to both multi-task learning and multi-granularity scenarios. In this work, we present \textbf{REF-VLM}, an end-to-end framework for unified training of various visual decoding tasks. To address complex visual decoding scenarios, we introduce the \textbf{Triplet-Based Referring Paradigm (TRP)}, which explicitly decouples three critical dimensions in visual decoding tasks through a triplet structure: concepts, decoding types, and targets. TRP employs symbolic delimiters to enforce structured representation learning, enhancing the parsability and interpretability of model outputs. Additionally, we construct \textbf{Visual-Task Instruction Following Dataset (VT-Instruct)}, a large-scale multi-task dataset containing over 100 million multimodal dialogue samples across 25 task types. Beyond text inputs and outputs, VT-Instruct incorporates various visual prompts such as point, box, scribble, and mask, and generates outputs composed of text and visual units like box, keypoint, depth and mask. The combination of different visual prompts and visual units generates a wide variety of task types, expanding the applicability of REF-VLM significantly. Both qualitative and quantitative experiments demonstrate that our REF-VLM outperforms other MLLMs across a variety of standard benchmarks. The code, dataset, and demo will be publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑