当前的视觉-语言多模态模型在一般视觉理解任务中表现良好,但由于缺乏专门的视觉-语言指令跟随数据,它们在处理与人体姿态和动作相关的复杂视觉任务时表现不佳。我们提出了一种通过将人体关键点与字幕、边界框等传统视觉特征相集成来生成此类数据的方法,从而能够更精确地理解以人为中心的场景。我们的方法构建了一个包含200,328个样本的数据集,专门用于微调面向以人为中心任务的模型,重点关注三个领域:对话、详细描述和复杂推理。我们建立了一个名为Human Pose and Action Understanding Benchmark(HPAUB)的基准,用于评估模型在人体姿态和动作理解方面的性能。我们使用该数据集对LLaVA-1.5-7B模型进行微调,并在该基准上进行评估,取得了显著改进。实验结果表明,与原始LLaVA-1.5-7B模型相比,整体性能提升了21.18%。这些发现凸显了关键点集成数据在增强多模态模型方面的有效性。代码可在https://github.com/Ody-trek/Keypoint-Instruction-Tuning获取。
英文摘要:
Current vision-language multimodal models are well-adapted for general visual understanding tasks. However, they perform inadequately when handling complex visual tasks related to human poses and actions due to the lack of specialized vision-language instruction-following data. We introduce a method for generating such data by integrating human keypoints with traditional visual features such as captions and bounding boxes, enabling more precise understanding of human-centric scenes. Our approach constructs a dataset comprising 200,328 samples tailored to fine-tune models for human-centric tasks, focusing on three areas: conversation, detailed description, and complex reasoning. We establish a benchmark called Human Pose and Action Understanding Benchmark (HPAUB) to assess model performance on human pose and action understanding. We fine-tune the LLaVA-1.5-7B model using this dataset and evaluate it on the benchmark, achieving significant improvements. Experimental results show an overall improvement of 21.18% compared to the original LLaVA-1.5-7B model. These findings highlight the effectiveness of keypoint-integrated data in enhancing multimodal models. Code is available at https://github.com/Ody-trek/Keypoint-Instruction-Tuning.
Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance
Harbin Institute of Technology(哈尔滨工业大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济发展实验室);Chongqing Research Institute of HIT(重庆哈工大研究院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学)
MGIMM: Multi-Granularity Instruction Multimodal Model for Attribute-Guided Remote Sensing Image Detailed Description
National Engineering Research Center for Multimedia Software(国家多媒体软件工程技术研究中心);School of Computer Science, Wuhan University(武汉大学计算机学院);Hubei Luojia Laboratory(湖北珞珈实验室)
SignIT: A Comprehensive Dataset and Multimodal Analysis for Italian Sign Language Recognition
Computer Science - University of Catania, Italy(计算机科学 - 卡塔尼亚大学,意大利);Next Vision s.r.l., Spin-off of the University of Catania, Italy(2 Next Vision s.r.l.,卡塔尼亚大学分校,意大利)
MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所多模态人工智能系统国家重点实验室);Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences(中国科学院香港创新科学研究院人工智能与机器人中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Fujian Key Laboratory of Pattern Recognition and Image Understanding, School of Computer and Information Engineering, Xiamen University of Technology(福建 pattern recognition and image understanding 工程学院,厦门大学科技学院)
MM-NeuroOnco: A Multimodal Benchmark and Instruction Dataset for MRI-Based Brain Tumor Diagnosis
Guangdong Institute of Intelligence Science and Technology(广东智能科学与技术研究院);Center for Brain-Inspired Computing Research (CBICR), Department of Precision Instrument, Tsinghua University(脑启发计算研究中心(CBICR)、精密仪器系,清华大学)
Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室)
Continual Instruction Tuning for Large Multimodal Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);The Peng Cheng Laboratory(鹏城实验室);Wuhan AI Research(武汉人工智能研究院)
MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment
CUHK MMLab(香港中文大学多媒体实验室);Waseda University(早稻田大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);SenseTime Research(商汤科技研究院);Shanghai AI Laboratory(上海人工智能实验室);CPII under InnoHK(创新香港旗下的CPII机构)
Can Multimodal LLMs See Science Instruction? Benchmarking Pedagogical Reasoning in K-12 Classroom Videos
Department of Computer Science, Drexel University(德雷塞尔大学计算机科学系);Department of Teaching and Learning, Washington State University(华盛顿州立大学教学与学习系);College of Chemistry, Beijing Normal University(北京师范大学化学学院);Department of Computer Science, University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校计算机科学系);Department of Data Science, City University of Hong Kong(香港城市大学数据科学系)
KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension
Sun Yat-sen University(中山大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));The University of Hong Kong(香港大学);The Chinese University of Hong Kong(香港中文大学);SenseTime Research(商汤科技研究院)
MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching
National Engineering Research Center for Multimedia Software(多媒体软件国家工程研究中心);Institute of Artificial Intelligence(人工智能研究院);School of Computer Science(计算机科学学院);Hubei Key Laboratory of Multimedia and Network Communication Engineering(湖北省多媒体与网络通信工程重点实验室);Lee Kong Chian School of Medicine(李光耀医学院);Nanyang Technological University(南洋理工大学);Ningbo Institute of Materials Technology and Engineering(宁波材料技术与工程研究所);Chinese Academy of Sciences(中国科学院);Institute for Infocomm Research (I 2 R)(信息与通信研究所以(I 2 R));Agency for Science, Technology and Research (A*STAR)(科技研究局(A*STAR))
MK-Pose: Category-Level Object Pose Estimation via Multimodal-Based Keypoint Learning
Institute of Robotics and Automatic Information System(机器人与自动信息系统研究所);Tianjin Key Laboratory of Intelligent Robotics(天津智能机器人重点实验室);TBI center(TBI中心);Nankai University(南开大学)
KptLLM++: Towards Generic Keypoint Comprehension with Large Language Model
Sun Yat-sen University, Shenzhen(中山大学深圳校区);Chinese University of Hong Kong, Shenzhen(香港中文大学深圳校区);SenseTime Research(商汤科技研究院);Chinese University of Hong Kong(香港中文大学);Guangdong Key Laboratory of Big Data Analysis and Processing(广东省大数据分析与处理重点实验室)
MIGE: Mutually Enhanced Multimodal Instruction-Based Image Generation and Editing
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
Investigating the Scaling Effect of Instruction Templates for Training Multimodal Language Model
Southeast University(东南大学);University of Southern California(南加州大学);University of Washington(华盛顿大学);University of California San Diego(加利福尼亚大学圣迭戈分校);ZOZO Research(ZOZO研究院);Xiaohongshu Inc(小红书公司);Peking University(北京大学);Waseda University(早稻田大学);Monash University(莫纳什大学)
ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(多模态人工智能系统国家重点实验室);School of Artificial Intelligence, UCAS(人工智能学院);Centre for Artificial Intelligence and Robotics, HKISI-CAS(人工智能与机器人中心)
SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models
University of Macau(澳门大学);ARC Lab, Tencent PCG(腾讯平台与内容事业群ARC实验室);Tencent AI Lab(腾讯人工智能实验室);Shenzhen Institute of Advanced Technology(深圳先进技术研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);Tsinghua University(清华大学)
IGD: Instructional Graphic Design with Multimodal Layer Generation
University of Science and Technology of China(中国科学技术大学);YuanShi Technology(元世科技);Institute of Trustworthy Embodied AI, Fudan University(复旦大学可信具身人工智能研究院)
MM-IFEngine: Towards Multimodal Instruction Following
Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Jiaotong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学);CPII under InnoHK(创新香港下的CPII);Shanghai Innovation Institute(上海创新研究院)
SilVar: Speech Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization
Florida Institute of Technology(佛罗里达理工学院);FPT University(FPT大学);Uppsala University(乌普萨拉大学);Knovel Engineering Lab(诺弗尔工程实验室);University of Alabama at Birmingham(阿拉巴马大学伯明翰分校)