Collaborative object localization aims to collaboratively estimate locations of objects observed from multiple views or perspectives, which is a critical ability for multi-agent systems such as connected vehicles. To enable collaborative localization, several model-based state estimation and learning-based localization methods have been developed. Given their encouraging performance, model-based state estimation often lacks the ability to model the complex relationships among multiple objects, while learning-based methods are typically not able to fuse the observations from an arbitrary number of views and cannot well model uncertainty. In this paper, we introduce a novel spatiotemporal graph filter approach that integrates graph learning and model-based estimation to perform multi-view sensor fusion for collaborative object localization. Our approach models complex object relationships using a new spatiotemporal graph representation and fuses multi-view observations in a Bayesian fashion to improve location estimation under uncertainty. We evaluate our approach in the applications of connected autonomous driving and multiple pedestrian localization. Experimental results show that our approach outperforms previous techniques and achieves the state-of-the-art performance on collaboration localization.
Is Intermediate Fusion All You Need for UAV-based Collaborative Perception?
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所认知与决策智能复杂系统重点实验室）
Leading autonomous vehicle (AV) platforms and testing infrastructures are, unfortunately, proprietary and closed-source. Thus, it is difficult to evaluate how well safety-critical AVs perform and how safe they truly are. Similarly, few platforms exist for much-needed multi-agent analysis. To provide a starting point for analysis of sensor fusion and collaborative & distributed sensing, we design an accessible, modular sensing platform with AVstack. We build collaborative and distributed camera-radar fusion algorithms and demonstrate an evaluation ecosystem of AV datasets, physics-based simulat…
CoBEVMoE: Heterogeneity-aware Feature Fusion with Dynamic Mixture-of-Experts for Collaborative Perception
School of Computer Science and Electronic Engineering, Hunan University（计算机科学与电子工程学院，湖南大学）；School of Artificial Intelligence and Robotics, Hunan University（人工智能与机器人学院，湖南大学）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（机器人视觉感知与控制技术国家工程研究中心，湖南大学）
IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection
School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；University of Macau（澳门大学）；The University of Hong Kong（香港大学）；Inceptio（嬴彻科技）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Zhejiang University（浙江大学）
School of Information, Renmin University of China（中国人民大学信息学院）；Sony Research and Development Center China（索尼（中国）研发有限公司）；National University of Defense Technology（国防科技大学）
Integrating Multi-Modal Sensors: A Review of Fusion Techniques for Intelligent Vehicles
College of Engineering, Center for Environmental Research and Technology, University of California at Riverside（工程学院、环境研究与技术中心、加州大学河滨分校）；School of Transportation and Logistics, Southwest Jiaotong University（交通运输与物流学院、西南交通大学）
EgoRefine: Ego-Referenced Predictive Alignment and Trajectory-Conditioned Reliability-Aware Fusion for Asynchronous Collaborative Perception
Hunan University（湖南大学）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（湖南大学机器人视觉感知与控制技术国家工程研究中心）；Nanjing University of Posts and Telecommunications（南京邮电大学）
Representation based classification methods have become a hot research topic during the past few years, and the two most prominent approaches are sparse representation based classification (SRC) and collaborative representation based classification (CRC). CRC reveals that it is the collaborative representation rather than the sparsity that makes SRC successful. Nevertheless, the dense representation of CRC may not be discriminative which will degrade its performance for classification tasks. To alleviate this problem to some extent, we propose a new method called sparse and collaborative-compe…
CoST: Efficient Collaborative Perception From Unified Spatiotemporal Perspective
Hangzhou International Innovation Institute, Beihang University（北京航空航天大学杭州国际创新研究院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；University of California, Los Angeles（美国加州大学洛杉矶分校）
Collaborative Attention and Consistent-Guided Fusion of MRI and PET for Alzheimer's Disease Diagnosis
School of Software Yunnan University（软件学院 云南大学）；Department of Computer Science The University of Sheffield（计算机科学系 剑桥大学）；Department of Computing Xian JiaoTong-Liverpool University（计算系 西交利物浦大学）
CoGoal3D: Collaborative 3D Object Detection with 3D-Aware Fusion and Refinement
Zhejiang University（浙江大学）；Zhejiang Provincial Key Laboratory of Multi-Modal Communication Networks and Intelligent Information Processing（浙江省多模态通信网络与智能信息处理重点实验室）；China Jiliang University（中国计量大学）
Integrating MedCLIP and Cross-Modal Fusion for Automatic Radiology Report Generation
The Bishop Strachan School（毕夏普斯特拉坎学校）；Baidu Inc.（百度公司）；Beihang University（北京航空航天大学）；Institute of Future Education（未来教育研究所）；TechArena Canada Inc.（加拿大泰克Arena公司）；College of Engineering（工程学院）；Northeastern University（东北大学）
Enhancing Visual In-Context Learning by Multi-Faceted Fusion
College of Intelligent Robotics and Advance Manufacturing, Fudan University（智能机器人与先进制造学院，复旦大学）；School of Microelectronics, Fudan University（微电子学院，复旦大学）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；School of Information Science and Engineering, East China University of Science and Technology（信息科学与工程学院，东华大学）
Universität Bremen（不来梅大学）；Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI)（德国人工智能研究中心）
Common estimation algorithms, such as least squares estimation or the Kalman filter, operate on a state in a state space S that is represented as a real-valued vector. However, for many quantities, most notably orientations in 3D, S is not a vector space, but a so-called manifold, i.e. it behaves like a vector space locally but has a more complex global topological structure. For integrating these quantities, several ad-hoc approaches have been proposed. Here, we present a principled solution to this problem where the structure of the manifold S is encapsulated by two operators, state displace…
CoLC: Communication-Efficient Collaborative Perception with LiDAR Completion
Key Laboratory of Big Data & Artificial Intelligence in Transportation, Ministry of Education（交通运输大数据与人工智能重点实验室，教育部）；School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）；Xiamen University（厦门大学）
Video caption refers to generating a descriptive sentence for a specific short video clip automatically, which has achieved remarkable success recently. However, most of the existing methods focus more on visual information while ignoring the synchronized audio cues. We propose three multimodal deep fusion strategies to maximize the benefits of visual-audio resonance information. The first one explores the impact on cross-modalities feature fusion from low to high order. The second establishes the visual-audio short-term dependency by sharing weights of corresponding front-end networks. The th…
Recently, table structure recognition has achieved impressive progress with the help of deep graph models. Most of them exploit single visual cues of tabular elements or simply combine visual cues with other modalities via early fusion to reason their graph relationships. However, neither early fusion nor individually reasoning in terms of multiple modalities can be appropriate for all varieties of table structures with great diversity. Instead, different modalities are expected to collaborate with each other in different patterns for different table cases. In the community, the importance of …
Sequence-to-sequence models have recently become very popular for tackling handwritten word recognition problems. However, how to effectively integrate an external language model into such recognizer is still a challenging problem. The main challenge faced when training a language model is to deal with the language model corpus which is usually different to the one used for training the handwritten word recognition system. Thus, the bias between both word corpora leads to incorrectness on the transcriptions, providing similar or even worse performances on the recognition task. In this work, we…
V2X-DSC: Multi-Agent Collaborative Perception with Distributed Source Coding Guided Communication
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）
Cross-Modal Action Recognition in Egocentric Video Using Mamba: Integrating RGB and Hand Skeleton Streams via CLS Token Fusion Strategies
Univ. of Buenos Aires. Faculty of Exact and Natural Sciences. Dept. of Computer Science (DC)（布宜诺斯艾利斯大学。精确与自然科学学院。计算机科学系）；CONICET-Univ. of Buenos Aires. Institute of Computer Sciences (ICC)（布宜诺斯艾利斯大学CONICET联合体。计算机科学研究所）
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学智能自主系统上海研究院）；School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；School of Mechatronic Engineering and Automation, Shanghai University（上海大学机械电子工程与自动化学院）
