In this paper, we address the problem of separating individual speech signals from videos using audio-visual neural processing. Most conventional approaches utilize frame-wise matching criteria to extract shared information between co-occurring audio and video. Thus, their performance heavily depends on the accuracy of audio-visual synchronization and the effectiveness of their representations. To overcome the frame discontinuity problem between two modalities due to transmission delay mismatch or jitter, we propose a cross-modal affinity network (CaffNet) that learns global correspondence as well as locally-varying affinities between audio and visual streams. Given that the global term provides stability over a temporal sequence at the utterance-level, this resolves the label permutation problem characterized by inconsistent assignments. By extending the proposed cross-modal affinity on the complex network, we further improve the separation performance in the complex spectral domain. Experimental results verify that the proposed methods outperform conventional ones on various datasets, demonstrating their advantages in real-world scenarios.
Exploring adaptation of VideoMAE for Audio-Visual Diarization & Social @ Ego4d Looking at me Challenge
Beijing University of Posts and Telecommunications（北京邮电大学）；Nanjing University（南京大学）
在本报告中，我们介绍了将预训练的视频掩码自动编码器（VideoMAE）转移到Ego 4d Looking at me Challenge的自我中心任务。VideoMAE是用于自我监督视频预训练的数据高效预训练模型，可以轻松地转移到下游任务。我们表明，从VideoMAE传输的表示具有良好的时空建模和捕捉小动作的能力。我们只需要使用以自我为中心的数据来训练基于VideoMAE的10个epoch，其中VideoMAE是由从第三人的视角获取的普通视频进行预训练的，并且我们可以得到比Ego 4d Looking at me Challenge的基线更好的结果。
Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Making
Machine Intelligence, Interaction, and Imagination (Mi3) Laboratory（机器智能、交互与想象实验室）；Laboratory for Intelligent and Safe Automobiles (LISA)（智能与安全汽车实验室）；Johns Hopkins University（约翰霍普金斯大学）；Center for Medicinal Cannabis Research (CMCR)（医药大麻研究中心）
ShizhenGPT: Towards Multimodal LLMs for Traditional Chinese Medicine
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Freedom AI；National Health Data Institute, Shenzhen（深圳国家健康数据研究院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Guangdong University of Technology（广东工业大学）
ByteNet: Rethinking Multimedia File Fragment Classification through Visual Perspectives
School of Electrical and Electronics Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）；School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University（香港理工大学电机工程学系）
