Jack of All Tasks, Master of Many: 设计通用的从粗到细视觉-语言模型
Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model
AI总结:
针对现有通用视觉-语言模型无法整合分割、多图像输入与粗粒度任务的问题,提出统一框架VistaLLM,结合指令引导图像分词器等技术,构建CoinIt数据集并提出AttCoSeg任务,在多类任务上取得最优性能。
摘要:
大语言模型(LLMs)处理视觉输入的能力催生了通用视觉系统,这类系统通过指令微调统一了各类视觉-语言(VL)任务。然而,由于视觉领域输入输出格式的巨大多样性,现有的通用模型无法成功将分割、多图像输入与粗粒度任务整合到单一框架中。本研究提出了VistaLLM,这是一个强大的视觉系统,可通过统一框架处理单张和多张输入图像上的粗粒度与细粒度VL任务。VistaLLM采用指令引导的图像分词器,利用任务描述过滤全局嵌入,从大量图像中提取经过压缩和优化的特征。此外,VistaLLM使用梯度感知自适应采样技术将二值分割掩码表示为序列,相比此前使用的均匀采样有显著提升。为增强VistaLLM的目标能力,我们构建了CoinIt,这是一个包含680万样本的综合性从粗到细指令微调数据集。我们还通过提出一项新任务AttCoSeg(属性级协同分割),解决了多图像定位数据集匮乏的问题,该任务提升了模型在多张输入图像上的推理与定位能力。在广泛的视觉(V)和视觉-语言(VL)任务上开展的大量实验证明了VistaLLM的有效性,它在所有下游任务上均优于强基线模型,取得了一致的最优性能。我们的项目页面位于https://shramanpramanick.github.io/VistaLLM/。
英文摘要:
The ability of large language models (LLMs) to process visual inputs has given rise to general-purpose vision systems, unifying various vision-language (VL) tasks by instruction tuning. However, due to the enormous diversity in input-output formats in the vision domain, existing general-purpose models fail to successfully integrate segmentation and multi-image inputs with coarse-level tasks into a single framework. In this work, we introduce VistaLLM, a powerful visual system that addresses coarse- and fine-grained VL tasks over single and multiple input images using a unified framework. VistaLLM utilizes an instruction-guided image tokenizer that filters global embeddings using task descriptions to extract compressed and refined features from numerous images. Moreover, VistaLLM employs a gradient-aware adaptive sampling technique to represent binary segmentation masks as sequences, significantly improving over previously used uniform sampling. To bolster the desired capability of VistaLLM, we curate CoinIt, a comprehensive coarse-to-fine instruction tuning dataset with 6.8M samples. We also address the lack of multi-image grounding datasets by introducing a novel task, AttCoSeg (Attribute-level Co-Segmentation), which boosts the model's reasoning and grounding capability over multiple input images. Extensive experiments on a wide range of V- and VL tasks demonstrate the effectiveness of VistaLLM by achieving consistent state-of-the-art performance over strong baselines across all downstream tasks. Our project page can be found at https://shramanpramanick.github.io/VistaLLM/.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。