增强视觉定位与泛化能力：一种面向视觉-语言模型的多任务循环训练方法
Enhancing Visual Grounding and Generalization: A Multi-Task Cycle Training Approach for Vision-Language Models
AI总结：
本文提出大型多模态模型ViLaM，通过REG与REC间的循环训练策略联合优化多任务视觉定位，显著提升模型在开放集、少样本及医学跨域场景下的泛化与交互能力，并开源相关数据集与代码。
摘要：
视觉定位（VG）在多模态视觉-语言模型中占据关键地位。在本研究中，我们提出了 ViLaM，一种大型多模态模型，该模型利用循环训练策略并辅以丰富的交互指令，支持 VG 的多任务处理。我们引入了指代表达生成（REG）与指代表达理解（REC）之间的循环训练。这增强了视觉定位与指代表达之间的一致性，并满足了对高质量、多任务 VG 数据集的需求。此外，得益于循环训练策略，我们的模型进一步促进了 VG 的多任务处理。REC 中的多任务涵盖了从区域级到像素级的多种粒度，包括指代边界框检测、指代关键点检测和指代图像分割。在 REG 中，指代区域分类确定目标的细粒度类别，而指代区域描述生成则生成全面的描述。同时，所有任务均参与联合训练，相互协同增强，共同提升模型的整体性能。此外，利用大型语言模型的能力，ViLaM 扩展了广泛的指令，从而显著增强了其泛化和交互潜力。大量公开数据集证实了我们的模型在多任务 VG 方面的卓越能力。此外，为了验证其鲁棒的泛化能力，ViLaM 在开放集和少样本场景下进行了验证。特别是在医学领域，我们的模型展现了跨域的鲁棒泛化能力。另外，我们贡献了一个 VG 数据集，特别是针对多任务设计的。为了支持和鼓励专注于 VG 的社区，我们已将数据集和代码公开：https://github.com/AnonymGiant/ViLaM。
英文摘要：
Visual grounding (VG) occupies a pivotal position in multi-modality vision-language models. In this study, we propose ViLaM, a large multi-modality model, that supports multi-tasks of VG using the cycle training strategy, with abundant interaction instructions. The cycle training between referring expression generation (REG) and referring expression comprehension (REC) is introduced. It enhances the consistency between visual location and referring expressions, and addresses the need for high-quality, multi-tasks VG datasets. Moreover, multi-tasks of VG are promoted in our model, contributed by the cycle training strategy. The multi-tasks in REC encompass a range of granularities, from region-level to pixel-level, which include referring bbox detection, referring keypoints detection, and referring image segmentation. In REG, referring region classification determines the fine-grained category of the target, while referring region captioning generates a comprehensive description. Meanwhile, all tasks participate in the joint training, synergistically enhancing one another and collectively improving the overall performance of the model. Furthermore, leveraging the capabilities of large language models, ViLaM extends a wide range of instructions, thereby significantly enhancing its generalization and interaction potentials. Extensive public datasets corroborate the superior capabilities of our model in VG with muti-tasks. Additionally, validating its robust generalization, ViLaM is validated under open-set and few-shot scenarios. Especially in the medical field, our model demonstrates cross-domain robust generalization capabilities. Furthermore, we contribute a VG dataset, especially with multi-tasks. To support and encourage the community focused on VG, we have made both the dataset and our code public: https://github.com/AnonymGiant/ViLaM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
