This work addresses the problem of sensing the world: how to learn a multimodal representation of a reinforcement learning agent's environment that allows the execution of tasks under incomplete perceptual conditions. To address such problem, we argue for hierarchy in the design of representation models and contribute with a novel multimodal representation model, MUSE. The proposed model learns hierarchical representations: low-level modality-specific representations, encoded from raw observation data, and a high-level multimodal representation, encoding joint-modality information to allow robust state estimation. We employ MUSE as the sensory representation model of deep reinforcement learning agents provided with multimodal observations in Atari games. We perform a comparative study over different designs of reinforcement learning agents, showing that MUSE allows agents to perform tasks under incomplete perceptual experience with minimal performance loss. Finally, we evaluate the performance of MUSE in literature-standard multimodal scenarios with higher number and more complex modalities, showing that it outperforms state-of-the-art multimodal variational autoencoders in single and cross-modality generation.
Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward
University of Science and Technology of China（中国科学技术大学）；State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）；iFLYTEK AI Research（iFLYTEK人工智能研究院）；Hong Kong University of Science and Technology（香港科学与技术大学）
Omni-Perception Policy Optimization for Multimodal Emotion Reasoning
University of Science and Technology of China（中国科学技术大学）；SenseTime Research（商汤科技研究院）；National University of Singapore（新加坡国立大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models
School of Computer Science, Sichuan University（四川大学计算机学院）；School of Electronic and Computer Engineering, Peking University Shenzhen Graduate School（北京大学深圳研究生院电子与计算机工程学院）；Institute of Artificial Intelligence (TeleAI), China Telecom and Northwestern Polytechnical University（中国电信与西北工业大学人工智能研究院（TeleAI））
Coupling Perception and Reasoning in Federated Multimodal Graph Foundation Models
Beijing Institute of Technology（北京理工大学）；Shandong University（山东大学）；Institute of Information Engineering（信息工程研究所）；University of Science and Technology Beijing（北京科技大学）
ECCV 2024 W-CODA: 1st Workshop on Multimodal Perception and Comprehension of Corner Cases in Autonomous Driving
Hong Kong University of Science and Technology（香港科学与技术大学）；The Chinese University of Hong Kong（香港中文大学）；Dalian University of Technology（大连理工大学）；TU Delft（代尔夫特理工大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Huawei Zurich Research Center（华为苏黎世研究中心）；Huawei IAS BU（华为IAS业务单元）
Walkable to Whom? Capturing Subjective Variability in Walkability Perception Using Multimodal Deep Learning
School of Civil and Environmental Engineering, University of New South Wales (UNSW)（新南威尔士大学土木与环境工程学院）；Research Centre for Integrated Transport Innovation (rCITI)（综合交通创新研究中心）
Waseda University（早稻田大学）；National Institute of Advanced Science and Technology（国立先进科学技术研究所）
Selection of appropriate tools and use of them when performing daily tasks is a critical function for introducing robots for domestic applications. In previous studies, however, adaptability to target objects was limited, making it difficult to accordingly change tools and adjust actions. To manipulate various objects with tools, robots must both understand tool functions and recognize object characteristics to discern a tool-object-action relation. We focus on active perception using multimodal sensorimotor data while a robot interacts with objects, and allow the robot to recognize their extr…
YOLO with Kolmogorov-Arnold networks and vision-language foundation models for interpretable object detection with trustworthy multimodal AI in computer vision perception
University of Bath（巴斯大学）；Zhejiang University（浙江大学）
Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation
State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences（机器人与智能系统国家重点实验室，沈阳自动化研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）
Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens
AIML, University of Adelaide（AIML，阿德莱德大学）；Monash University（墨尔本大学）；Australian National University（澳大利亚国立大学）；Tiktok, Australia（Tiktok，澳大利亚）；Zhejiang University（浙江大学）
