arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.09306cs.CV
收藏

关键点集成的指令跟随数据生成方法用于增强多模态模型的人体姿态与动作理解

Keypoint-Integrated Instruction-Following Data Generation for Enhanced Human Pose and Action Understanding in Multimodal Models

Department of Informatics, Graduate School of Informatics and Engineering, The University of Electro-Communications(信息学院、信息与工程研究生院、电通通信大学)

Dewen Zhang, Wangpeng An, Hayaru Shouno

AI总结:

本文提出将人体关键点与字幕、边界框等视觉特征集成以生成指令跟随数据,构建20万余样本数据集并建立HPAUB基准,微调LLaVA-1.5-7B后整体性能提升21.18%。

摘要:

当前的视觉-语言多模态模型在一般视觉理解任务中表现良好,但由于缺乏专门的视觉-语言指令跟随数据,它们在处理与人体姿态和动作相关的复杂视觉任务时表现不佳。我们提出了一种通过将人体关键点与字幕、边界框等传统视觉特征相集成来生成此类数据的方法,从而能够更精确地理解以人为中心的场景。我们的方法构建了一个包含200,328个样本的数据集,专门用于微调面向以人为中心任务的模型,重点关注三个领域:对话、详细描述和复杂推理。我们建立了一个名为Human Pose and Action Understanding Benchmark(HPAUB)的基准,用于评估模型在人体姿态和动作理解方面的性能。我们使用该数据集对LLaVA-1.5-7B模型进行微调,并在该基准上进行评估,取得了显著改进。实验结果表明,与原始LLaVA-1.5-7B模型相比,整体性能提升了21.18%。这些发现凸显了关键点集成数据在增强多模态模型方面的有效性。代码可在https://github.com/Ody-trek/Keypoint-Instruction-Tuning获取。

英文摘要:

Current vision-language multimodal models are well-adapted for general visual understanding tasks. However, they perform inadequately when handling complex visual tasks related to human poses and actions due to the lack of specialized vision-language instruction-following data. We introduce a method for generating such data by integrating human keypoints with traditional visual features such as captions and bounding boxes, enabling more precise understanding of human-centric scenes. Our approach constructs a dataset comprising 200,328 samples tailored to fine-tune models for human-centric tasks, focusing on three areas: conversation, detailed description, and complex reasoning. We establish a benchmark called Human Pose and Action Understanding Benchmark (HPAUB) to assess model performance on human pose and action understanding. We fine-tune the LLaVA-1.5-7B model using this dataset and evaluate it on the benchmark, achieving significant improvements. Experimental results show an overall improvement of 21.18% compared to the original LLaVA-1.5-7B model. These findings highlight the effectiveness of keypoint-integrated data in enhancing multimodal models. Code is available at https://github.com/Ody-trek/Keypoint-Instruction-Tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑