aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range Perception
aiMotive
Tamás Matuszka, Iván Barton, Ádám Butykai, Péter Hajas, Dávid Kiss, Domonkos Kovács, Sándor Kunsági-Máté, Péter Lengyel, Gábor Németh, Levente Pető, Dezső Ribli… 展开作者
Tamás Matuszka, Iván Barton, Ádám Butykai, Péter Hajas, Dávid Kiss, Domonkos Kovács, Sándor Kunsági-Máté, Péter Lengyel, Gábor Németh, Levente Pető, Dezső Ribli, Dávid Szeghy, Szabolcs Vajna, Bálint Varga
备注：
The paper was accepted to ICLR 2023 Workshop Scene Representations for Autonomous Driving
本文提出了 aiMotive 多模态自动驾驶数据集，包含 176 个场景的同步 LiDAR、相机和雷达数据，覆盖多种天气与光照条件并提供远距离 3D 标注，同时训练了单模态和多模态 3D 目标检测基线模型。
摘要：
自动驾驶是计算机视觉研究领域中的一个热门研究方向。由于自动驾驶车辆具有极高的安全关键性，确保鲁棒性对于实际部署至关重要。虽然已有多个公开的多模态数据集可供使用，但它们主要包含两种传感器模态（相机、LiDAR），并不适合恶劣天气条件。此外，这些数据集缺乏远距离标注，使得训练作为自动驾驶车辆高速公路辅助功能基础的神经网络变得更加困难。因此，我们引入了一个面向具有远距离感知的鲁棒自动驾驶的多模态数据集。该数据集包含 176 个场景，配备同步且已标定的 LiDAR、相机和雷达传感器，覆盖 360 度视场。采集的数据来自高速公路、城市和郊区区域，涵盖白天、夜间和雨天，并标注了跨帧具有一致标识符的 3D 边界框。此外，我们训练了用于 3D 目标检测的单模态和多模态基线模型。数据可在 \url{https://github.com/aimotive/aimotive_dataset} 获取。
英文摘要：
Autonomous driving is a popular research area within the computer vision research community. Since autonomous vehicles are highly safety-critical, ensuring robustness is essential for real-world deployment. While several public multimodal datasets are accessible, they mainly comprise two sensor modalities (camera, LiDAR) which are not well suited for adverse weather. In addition, they lack far-range annotations, making it harder to train neural networks that are the base of a highway assistant function of an autonomous vehicle. Therefore, we introduce a multimodal dataset for robust autonomous driving with long-range perception. The dataset consists of 176 scenes with synchronized and calibrated LiDAR, camera, and radar sensors covering a 360-degree field of view. The collected data was captured in highway, urban, and suburban areas during daytime, night, and rain and is annotated with 3D bounding boxes with consistent identifiers across frames. Furthermore, we trained unimodal and multimodal baseline models for 3D object detection. Data are available at \url{https://github.com/aimotive/aimotive_dataset}.
Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulation
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；School of Psychological and Cognitive Sciences, Peking University（北京大学心理与认知科学学院）；Beijing Key Lab of Behavior and Mental Health, Peking University（北京大学行为与心理健康北京市重点实验室）；Beijing Institute for General Artificial Intelligence（北京一般人工智能研究院）；State Key Lab for General Artificial Intelligence（一般人工智能国家重点实验室）；Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence（具身智能实验室，北京大学武汉人工智能研究院）；Department of Computer Science and Technology, University of Cambridge（剑桥大学计算机科学与技术系）
ECCV 2024 W-CODA: 1st Workshop on Multimodal Perception and Comprehension of Corner Cases in Autonomous Driving
Hong Kong University of Science and Technology（香港科学与技术大学）；The Chinese University of Hong Kong（香港中文大学）；Dalian University of Technology（大连理工大学）；TU Delft（代尔夫特理工大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Huawei Zurich Research Center（华为苏黎世研究中心）；Huawei IAS BU（华为IAS业务单元）
University of Southern California（南加州大学）；U.S. Army Research Laboratory（美国陆军研究实验室）
Current perception systems often carry multimodal imagers and sensors such as 2D cameras and 3D LiDAR sensors. To fuse and utilize the data for downstream perception tasks, robust and accurate calibration of the multimodal sensor data is essential. We propose a novel deep learning-driven technique (CalibDNN) for accurate calibration among multimodal sensor, specifically LiDAR-Camera pairs. The key innovation of the proposed work is that it does not require any specific calibration targets or hardware assistants, and the entire processing is fully automatic with a single model and single iterat…
Due to the high complexity and occlusion, insufficient perception in the crowded urban intersection can be a serious safety risk for both human drivers and autonomous algorithms, whereas CVIS (Cooperative Vehicle Infrastructure System) is a proposed solution for full-participants perception in this scenario. However, the research on roadside multimodal perception is still in its infancy, and there is no open-source dataset for such scenario. Accordingly, this paper fills the gap. Through an IPS (Intersection Perception System) installed at the diagonal of the intersection, this paper proposes …
GPT-4o: Visual perception performance of multimodal large language models in piglet activity understanding
China Agricultural University（中国农业大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；AniEye Technology（安尼眼科技）；State Key Laboratory of Animal Nutrition and Feeding（动物营养与饲料国家重点实验室）
Spotlight on Token Perception for Multimodal Reinforcement Learning
Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；The Chinese University of Hong Kong（香港中文大学）；Nanjing University（南京大学）；Wuhan University（武汉大学）
VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception
Zhejiang University（浙江大学）；Shanghai AI Laboratory（上海人工智能实验室）；Nanjing University（南京大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学智能自主系统上海研究院）；School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；School of Mechatronic Engineering and Automation, Shanghai University（上海大学机械电子工程与自动化学院）
FaceInsight: A Multimodal Large Language Model for Face Perception
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区计算机科学与技术学院）；Key Laboratory of Education Informatization for Nationalities (Yunnan Normal University), Ministry of Education（民族教育信息化重点实验室（云南师范大学））
Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward
University of Science and Technology of China（中国科学技术大学）；State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）；iFLYTEK AI Research（iFLYTEK人工智能研究院）；Hong Kong University of Science and Technology（香港科学与技术大学）
ShanghaiTech University（上海科技大学）；The Chinese University of Hong Kong（香港中文大学）；RWTH Aachen University（亚琛工业大学）；Shanghai AI Laboratory（上海人工智能实验室）；University of Kentucky（肯塔基大学）；University of Maryland at College Park（马里兰大学帕克分校）
Accurately detecting and tracking pedestrians in 3D space is challenging due to large variations in rotations, poses and scales. The situation becomes even worse for dense crowds with severe occlusions. However, existing benchmarks either only provide 2D annotations, or have limited 3D annotations with low-density pedestrian distribution, making it difficult to build a reliable pedestrian perception system especially in crowded scenes. To better evaluate pedestrian perception algorithms in crowded scenarios, we introduce a large-scale multimodal dataset,STCrowd. Specifically, in STCrowd, there…
Reinforcing Egocentric Spatial Perception in Multimodal Large Language Models via Ego Scene Augmentation
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Guangxi Zhuang Autonomous Region Information Center（广西壮族自治区信息中心）；The Hong Kong University of Science and Technology（香港科技大学）
Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）；Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd（星辰通用人工智能实验室，中国电信人工智能技术（北京）有限公司）；University of Science and Technology Beijing（北京科技大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Jiao Tong University（上海交通大学）
DifferSeg: Towards Diverse Multimodal Binary Segmentation via Differential Perception and Frequency Guidance
School of Artificial Intelligence, Jiangxi Normal University（江西师范大学人工智能学院）；Institute of AI Education, East China Normal University（华东师范大学人工智能教育研究所）；Yale School of Medicine, Yale University（耶鲁大学医学院）
Humans usually perceive the world in a multimodal way that vision, touch, sound are utilised to understand surroundings from various dimensions. These senses are combined together to achieve a synergistic effect where the learning is more effectively than using each sense separately. For robotics, vision and touch are two key senses for the dexterous manipulation. Vision usually gives us apparent features like shape, color, and the touch provides local information such as friction, texture, etc. Due to the complementary properties between visual and tactile senses, it is desirable for us to co…
