Human-scene vision-language tasks are increasingly prevalent in diverse social applications, yet recent advancements predominantly rely on models specifically tailored to individual tasks. Emerging research indicates that large vision-language models (VLMs) can enhance performance across various downstream vision-language understanding tasks. However, general-domain models often underperform in specialized fields. This study introduces a domain-specific Large Vision-Language Model, Human-Scene Vision-Language Model (HumanVLM), designed to provide a foundation for human-scene Vision-Language tasks. Specifically, (1) we create a large-scale human-scene multimodal image-text dataset (HumanCaption-10M) sourced from the Internet to facilitate domain-specific alignment; (2) develop a captioning approach for human-centered images, capturing human faces, bodies, and backgrounds, and construct a high-quality Human-Scene image-text dataset (HumanCaptionHQ, about 311k pairs) that contain as much detailed information as possible about human; (3) Using HumanCaption-10M and HumanCaptionHQ, we train a HumanVLM. In the experiments, we then evaluate our HumanVLM across varous downstream tasks, where it demonstrates superior overall performance among multimodal models of comparable scale, particularly excelling in human-related tasks and significantly outperforming similar models, including Qwen2VL and ChatGPT-4o. HumanVLM, alongside the data introduced, will stimulate the research in human-around fields.
Argus: Leveraging Multiview Images for Improved 3-D Scene Understanding With Large Language Models
School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);Beijing Digital Native Digital City Research Center(北京数字原生数字城市研究院);School of Computing, The University of Georgia(佐治亚大学计算机学院);School of Computer and Communication Engineering, University of Science and Technology Beijing(北京科技大学计算机与通信工程学院);Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学(深圳)计算机科学与工程系)
Foundation models on the bridge: Semantic hazard detection and safety maneuvers for maritime autonomy with vision-language models
Dept. of Mechanical and Industrial Engineering, NTNU(机械与工业工程系,挪威科技大学);Dept. of Aeronautics and Astronautics, Stanford University(航空航天工程系,斯坦福大学);Dept. of Computer Science, Stanford University(计算机科学系,斯坦福大学);NVIDIA Research(NVIDIA研究)
DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features
NVIDIA Research(英伟达研究院);University of Toronto(多伦多大学);University of Southern California(南加州大学);National University of Singapore(新加坡国立大学);Stanford University(斯坦福大学)
RA-LWLM: Retrieval-Augmented In-Context Localization with Wireless Foundation Models
Department of Electrical Engineering, Chalmers University of Technology(查尔姆斯理工大学电子工程系);Department of Electrical and Computer Engineering, Aarhus University(阿鲁斯大学电子与计算机工程系)
Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenes
McGovern Institute for Brain Research, MIT(麻省理工学院麦戈文脑研究所);Reality Labs, Meta(Meta现实实验室);Department of Brain and Cognitive Sciences, MIT(麻省理工学院脑与认知科学系);Department of Electrical Engineering and Computer Science, MIT(麻省理工学院电气工程与计算机科学系)
TasselNetV4: A vision foundation model for cross-scene, cross-scale, and cross-species plant counting
National Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation(multispectral information intelligent processing technology, school of artificial intelligence and automation);China-Poland Joint Laboratory on Measurement and Control Technology, School of Artificial Intelligence and Automation;School of Cyber Science and Engineering;School of Computer Science and Engineering;Wuhan Institute of Technology;Graduate School of Agricultural and Life Sciences;Engineering Research Center of Plant Phenotyping, Ministry of Education, Jiangsu Collaborative Innovation Center for Modern Crop Production, Academy for Advanced Interdisciplinary Studies
Grounding Foundational Vision Models with 3D Human Poses for Robust Action Recognition
Arizona State University(亚利桑那州立大学);Emory University(埃默里大学);University of California, Berkeley(加州大学伯克利分校);Delft University of Technology(代尔夫特理工大学);Algoverse AI Research(Algoverse AI 研究)