Letian Fu, Gaurav Datta, Huang Huang, William Chung-Ho Panitch, Jaimyn Drake, Joseph Ortiz, Mustafa Mukadam, Mike Lambeta, Roberto Calandra, Ken Goldberg
Touch is an important sensing modality for humans, but it has not yet been incorporated into a multimodal generative language model. This is partially due to the difficulty of obtaining natural language labels for tactile data and the complexity of aligning tactile readings with both visual observations and language descriptions. As a step towards bridging that gap, this work introduces a new dataset of 44K in-the-wild vision-touch pairs, with English language labels annotated by humans (10%) and textual pseudo-labels from GPT-4V (90%). We use this dataset to train a vision-language-aligned tactile encoder for open-vocabulary classification and a touch-vision-language (TVL) model for text generation using the trained encoder. Results suggest that by incorporating touch, the TVL model improves (+29% classification accuracy) touch-vision-language alignment over existing models trained on any pair of those modalities. Although only a small fraction of the dataset is human-labeled, the TVL model demonstrates improved visual-tactile understanding over GPT-4V (+12%) and open-source vision-language models (+32%) on a new touch-vision understanding benchmark. Code and data: https://tactile-vlm.github.io.
University of Cambridge（剑桥大学）；University of Glasgow（格拉斯哥大学）
Infants' neurological development is heavily influenced by their motor skills. Evaluating a baby's movements is key to understanding possible risks of developmental disorders in their growth. Previous research in psychology has shown that measuring specific movements or gestures such as face touches in babies is essential to analyse how babies understand themselves and their context. This research proposes the first automatic approach that detects face touches from video recordings by tracking infants' movements and gestures. The study uses a multimodal feature fusion approach mixing spatial a…
PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，中国科学院自动化研究所）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Data Science and Artificial Intelligence Research Institute, China United Network Communications Group Co., Ltd.（中国联合网络通信集团有限公司数据科学与人工智能研究院）
MAGE: Multimodal Alignment and Generation Enhancement via Bridging Visual and Semantic Spaces
Global Tone Communication Technology Co., Ltd.（全球 tone 通信技术有限公司）；Faculty of computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Seoul National University（首尔国立大学）；University of Washington（华盛顿大学）
Large language models readily adapt to novel settings, even without task-specific training data. Can their zero-shot capacity be extended to multimodal inputs? In this work, we propose ESPER which extends language-only zero-shot models to unseen multimodal tasks, like image and audio captioning. Our key novelty is to use reinforcement learning to align multimodal inputs to language model generations without direct supervision: for example, in the image case our reward optimization relies only on cosine similarity derived from CLIP, and thus requires no additional explicitly paired (image, capt…
Humans usually perceive the world in a multimodal way that vision, touch, sound are utilised to understand surroundings from various dimensions. These senses are combined together to achieve a synergistic effect where the learning is more effectively than using each sense separately. For robotics, vision and touch are two key senses for the dexterous manipulation. Vision usually gives us apparent features like shape, color, and the touch provides local information such as friction, texture, etc. Due to the complementary properties between visual and tactile senses, it is desirable for us to co…
Seeing Through Touch: Tactile-Driven Visual Localization of Material Regions
Korea Advanced Institute of Science and Technology（韩国科学技术院）；Hankuk University of Foreign Studies（韩国外国语大学）；Ulsan National Institute of Science and Technology（蔚山科学技术院）
Large Language Model Guided Progressive Feature Alignment for Multimodal UAV Object Detection
School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；University of Science and Technology of China（中国科学技术大学）
SGANet: Semantic and Geometric Alignment for Multimodal Multi-view Anomaly Detection
Smart Manufacturing Thrust, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）智能制造学域）；College of Mechanical and Vehicle Engineering, Hunan University（湖南大学机械与运载工程学院）；The Hong Kong University of Science and Technology（香港科技大学）
DashFusion: Dual-stream Alignment with Hierarchical Bottleneck Fusion for Multimodal Sentiment Analysis
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Beijing National Research Center for Information Science and Technology, Tsinghua University（清华大学信息科学与技术国家研究中心）；Department of Automation, Tsinghua University（清华大学自动化系）
CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillation
Peking University（北京大学）；Shenzhen Graduate School（深圳研究生院）；University of Science and Technology of China（中国科学技术大学）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Peking University（北京大学）；Northeastern University（东北大学）；Sydney Smart Technology College（悉尼智能技术学院）；School of Computer Science, Peking University（北京大学计算机学院）；The State Key Laboratory of Multimedia Information Processing（多媒体信息处理国家重点实验室）
Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Zhongke Fanyu Technology Co., Ltd（中科泛语科技有限公司）
LVRPO: Language-Visual Alignment with GRPO for Multimodal Understanding and Generation
Department of Machine Learning, CMU, USA（卡内基梅隆大学机器学习系，美国）；Department of Machine Learning, MBZUAI, UAE（穆罕默德·本·扎耶德人工智能大学机器学习系，阿联酋）；Department of Artificial Intelligence, Hanyang University ERICA, South Korea（汉阳大学ERICA校区人工智能系，韩国）
Dynamic Alignment and Calibration for Multimodal Learning
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Tianyijiaotong Technology Ltd.（天翼交通科技有限公司）；School of Computer Science and Technology, Hainan University（海南大学计算机科学与技术学院）
Progressive Multimodal Alignment for Continual Instruction Tuning
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）；Kyoto University（京都大学）；Migu Culture Technology Co.,Ltd.（咪咕文化科技有限公司）；State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology（脑认知与类脑智能技术国家重点实验室）
MaViLS, a Benchmark Dataset for Video-to-Slide Alignment, Assessing Baseline Accuracy with a Multimodal Alignment Algorithm Leveraging Speech, OCR, and Visual Features
Karlsruhe University of Applied Sciences（卡尔斯鲁厄应用技术大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；University of Hohenheim（霍恩海姆大学）
MemoCare: An Interactive Multimodal Mobile System for Automated Cognitive Screening
Lausanne University Hospital（洛桑大学医院）；Faculty of Biology and Medicine, University of Lausanne（洛桑大学生物与医学学院）；VNU University of Engineering and Technology（越南国立大学工程技术大学）；VinUni-Illinois Smart Health Center, VinUniversity（文大-伊利诺伊智能健康中心，文大）；University of Science, VNU-HCM（胡志明市越南国立大学科学大学）；Hanoi Medical University（河内医科大学）；National Geriatric Hospital（国家老年病医院）；Military Hospital 175（第175军医院）；School of Medicine, University of Medicine and Pharmacy at Ho Chi Minh City（胡志明市医药大学医学院）；International University, VNU-HCM（胡志明市越南国立大学国际大学）；Vietnam National University Ho Chi Minh City（胡志明市越南国立大学）
BRIDGE: Multimodal-to-Text Retrieval via Reinforcement-Learned Query Alignment
High institute for computer & information systems（高等计算机与信息系统学院）；Chungbuk National University（忠北大学）；Assiut University（艾斯尤特大学）；University of Innsbruck（因斯布鲁克大学）
When Alignment Fails: Multimodal Adversarial Attacks on Vision-Language-Action Models
TGAI Lab, School of Engineering, Westlake University（西拉库大学工程学院TGAI实验室）；Pennsylvania State University（宾夕法尼亚州立大学）；Sony Research, Sony（索尼研究实验室，索尼）；Xidian University（西安电子科技大学）
