Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval
Goethe University Frankfurt（法兰克福歌德大学）；Columbia University（哥伦比亚大学）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；IBM Research AI（IBM研究院人工智能部门）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；UT Austin（德克萨斯大学奥斯汀分校）
Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Rogerio Feris, David Harwath, James Glass, Hilde Kuehne
备注：
CVPR2022. The final published version of the proceedings will be available on IEEE Xplore
Multi-modal learning from video data has seen increased attention recently as it allows to train semantically meaningful embeddings without human annotation enabling tasks like zero-shot retrieval and classification. In this work, we present a multi-modal, modality agnostic fusion transformer approach that learns to exchange information between multiple modalities, such as video, audio, and text, and integrate them into a joined multi-modal representation to obtain an embedding that aggregates multi-modal temporal information. We propose to train the system with a combinatorial loss on everything at once, single modalities as well as pairs of modalities, explicitly leaving out any add-ons such as position or modality encoding. At test time, the resulting model can process and fuse any number of input modalities. Moreover, the implicit properties of the transformer allow to process inputs of different lengths. To evaluate the proposed approach, we train the model on the large scale HowTo100M dataset and evaluate the resulting embedding space on four challenging benchmark datasets obtaining state-of-the-art results in zero-shot video retrieval and zero-shot video action localization.
Knowledge-Informed Multi-Agent Trajectory Prediction at Signalized Intersections for Infrastructure-to-Everything
College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）；Shanghai Institute of Intelligent Science and Technology, Tongji University（同济大学上海智能科学与技术研究院）；Department of Electrical and Computer Engineering, Technical University of Munich（慕尼黑技术大学电子与计算机工程系）
HeCoFuse: Cross-Modal Complementary V2X Cooperative Perception with Heterogeneous Sensors
College of Engineering, Center for Environmental Research and Technology, University of California at Riverside（加州大学河滨分校工程学院、环境研究与技术中心）；School of Transportation and Logistics, Southwest Jiaotong University（西南交通大学交通运输与物流学院）；Chair of Robotics, Artificial Intelligence and Real-time Systems, TUM School of Computation, Information and Technology, Technical University of Munich（慕尼黑技术大学计算机、信息与技术学院机器人、人工智能与实时系统教授职位）
Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning
Southern University of Science and Technology（南方科技大学）；The Hong Kong University of Science and Technology（香港科技大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Huawei Cloud Project（华为云项目）
Align3D-AD: Cross-Modal Feature Alignment and Dual-Prompt Learning for Zero-shot 3D Anomaly Detection
Smart Manufacturing Thrust, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）智能制造方向）；The Hong Kong University of Science and Technology（香港科技大学）；College of Mechanical and Vehicle Engineering, Hunan University（湖南大学机械与车辆工程学院）
Energy-Latency Manipulation of Multi-modal Large Language Models via Verbose Samples
Tsinghua University（清华大学）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Peng Cheng Laboratory（鹏城实验室）；University of Oxford（牛津大学）；Tencent Technology (Beijing) Co.Ltd（腾讯科技（北京）有限公司）；Tencent Data Platform（腾讯数据平台）
DriverMHG: A Multi-Modal Dataset for Dynamic Recognition of Driver Micro Hand Gestures and a Real-Time Recognition Framework
TU Munich（慕尼黑工业大学）；LMU Munich（慕尼黑大学）；University of Tuebingen（图宾根大学）；Intel Labs Europe（英特尔欧洲实验室）；Intel Deutschland GmbH（英特尔德国有限公司）
手势的使用为人机交互（HCI）系统的笨重接口设备提供了自然的替代方案。然而，从视频流中实时的【形】动态微手势的识别对于车载场景是具有挑战性的，因为（i）手势应该自然地执行而不会分散驾驶员的注意力，（ii）微手势在空间受限区域处发生在非常短的时间间隔内，（iii）所执行的手势应该仅被识别一次，及（iv）整个架构应设计为轻量型，因为其将部署到嵌入式系统。在这项工作中，我们提出了一个HCI系统的动态识别驾驶员的微手势，这可能会产生至关重要的影响，特别是在汽车行业的安全相关的问题。为此，我们最初收集了一个名为Driver Micro Hand Gestures（DriverMHG）的数据集，该数据集由RGB、深度和红外模态组成。微手势的动态识别的挑战已经解决了提出一个轻量级的卷积神经网络（CNN）的基础架构，有效地运行在网上与滑动窗口的方法。对于CNN模型，应用了几种三维资源有效网络，并分析了它们的性能。手势的在线识别已与3D-MobileNetV 2，它提供了最好的离线准确性之间的应用网络具有类似的计算复杂性。最终的架构部署在驾驶员模拟器上，实时的【形】操作。我们公开了DriverMHG数据集和我们的源代码。
