arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.03034cs.AIcs.MM
收藏

HumanVLM:人类-场景视觉语言模型的基础

HumanVLM: Foundation for Human-Scene Vision-Language Model

Chongqing University of Posts and Telecommunications(重庆邮电大学)

Dawei Dai, Xu Long, Li Yutang, Zhang Yuanhui, Shuyin Xia

AI总结:

本文提出人类-场景视觉语言模型HumanVLM,构建大规模数据集并训练模型,在人类相关任务上表现优异,超越同类模型。

摘要:

人类-场景视觉语言任务在各种社会应用中日益普遍,然而最近的进展主要依赖于针对单个任务专门定制的模型。新兴研究表明,大型视觉语言模型(VLMs)可以提升各种下游视觉语言理解任务的性能。然而,通用领域的模型在专业领域往往表现不佳。本研究引入了一个特定领域的大型视觉语言模型——人类-场景视觉语言模型(HumanVLM),旨在为人类-场景视觉语言任务提供基础。具体而言,(1)我们创建了一个大规模的人类-场景多模态图像-文本数据集(HumanCaption-10M),该数据集来源于互联网,以促进特定领域的对齐;(2)开发了一种针对以人为中心的图像的描述方法,捕捉人脸、人体和背景,并构建了一个高质量的人类-场景图像-文本数据集(HumanCaptionHQ,约311k对),其中包含尽可能多的人类详细信息;(3)使用HumanCaption-10M和HumanCaptionHQ,我们训练了HumanVLM。在实验中,我们在各种下游任务上评估了我们的HumanVLM,结果表明,在可比规模的多模态模型中,它展现了优越的整体性能,特别是在与人类相关的任务中表现出色,并显著优于类似模型,包括Qwen2VL和ChatGPT-4o。HumanVLM及其引入的数据将促进人类周边领域的研究。

英文摘要:

Human-scene vision-language tasks are increasingly prevalent in diverse social applications, yet recent advancements predominantly rely on models specifically tailored to individual tasks. Emerging research indicates that large vision-language models (VLMs) can enhance performance across various downstream vision-language understanding tasks. However, general-domain models often underperform in specialized fields. This study introduces a domain-specific Large Vision-Language Model, Human-Scene Vision-Language Model (HumanVLM), designed to provide a foundation for human-scene Vision-Language tasks. Specifically, (1) we create a large-scale human-scene multimodal image-text dataset (HumanCaption-10M) sourced from the Internet to facilitate domain-specific alignment; (2) develop a captioning approach for human-centered images, capturing human faces, bodies, and backgrounds, and construct a high-quality Human-Scene image-text dataset (HumanCaptionHQ, about 311k pairs) that contain as much detailed information as possible about human; (3) Using HumanCaption-10M and HumanCaptionHQ, we train a HumanVLM. In the experiments, we then evaluate our HumanVLM across varous downstream tasks, where it demonstrates superior overall performance among multimodal models of comparable scale, particularly excelling in human-related tasks and significantly outperforming similar models, including Qwen2VL and ChatGPT-4o. HumanVLM, alongside the data introduced, will stimulate the research in human-around fields.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑