多模态大模型

The Multimodal Information based Speech Processing (MISP) 2022 Challenge: Audio-Visual Diarization and Recognition

Zhe Wang, Shilong Wu, Hang Chen, Mao-Kui He, Jun Du, Chin-Hui Lee, Jingdong Chen, Shinji Watanabe, Sabato Siniscalchi, Odette Scharenborg, Diyuan Liu, Baocai Yin, Jia Pan, Jianqing Gao, Cong Liu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(title)；multi-modal(abstract)；分类 cs.MM

Comments 5 pages, 4 figures, to be published in ICASSP2023

Self-Supervised Audio-Visual Speech Representations Learning By Multimodal Self-Distillation

Jing-Xuan Zhang, Genshun Wan, Zhen-Hua Ling, Jia Pan, Jianqing Gao, Cong Liu

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Comments submitted to ICASSP 2023

Self-Supervised Audio-Visual Representation Learning with Relaxed Cross-Modal Synchronicity

Pritam Sarkar, Ali Etemad

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

Comments Accepted in AAAI 2023

A Novel Frame Structure for Cloud-Based Audio-Visual Speech Enhancement in Multimodal Hearing-aids

Abhijeet Bishnu, Ankit Gupta, Mandar Gogate, Kia Dashtipour, Ahsan Adeel, Amir Hussain, Mathini Sellathurai, Tharmalingam Ratnarajah

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Multimodal Audio-Visual Information Fusion using Canonical-Correlated Graph Neural Network for Energy-Efficient Speech Enhancement

Leandro Aparecido Passos, João Paulo Papa, Javier Del Ser, Amir Hussain, Ahsan Adeel

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking

Yidi Li, Hong Liu, Hao Tang

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(title)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by AAAI2022

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Donghuo Zeng, Jianming Wu, Gen Hattori, Yi Yu, Rong Xu

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM

A Survey on Audio Synthesis and Audio-Visual Multimodal Processing

Zhaofeng Shi

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Attention-based cross-modal fusion for audio-visual voice activity detection in musical video streams

Yuanbo Hou, Zhesong Yu, Xia Liang, Xingjian Du, Bilei Zhu, Zejun Ma, Dick Botteldooren

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Comments Accepted by INTERSPEECH 2021

Rethinking the constraints of multimodal fusion: case study in Weakly-Supervised Audio-Visual Video Parsing

Jianning Wu, Zhuqing Jiang, Shiping Wen, Aidong Men, Haiying Wang

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency

Ruohan Gao, Kristen Grauman

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

Comments In CVPR 2021. Project page: http://vision.cs.utexas.edu/projects/VisualVoice/

Audio-Visual Instance Discrimination with Cross-Modal Agreement

Pedro Morgado, Nuno Vasconcelos, Ishan Misra

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

Audio-Visual Speech Separation Using Cross-Modal Correspondence Loss

Naoki Makishima, Mana Ihori, Akihiko Takashima, Tomohiro Tanaka, Shota Orihashi, Ryo Masumura

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Comments Accepted to ICASSP 2021

Audio-visual Speaker Recognition with a Cross-modal Discriminative Network

Ruijie Tao, Rohan Kumar Das, Haizhou Li

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Multimodal Integration for Large-Vocabulary Audio-Visual Speech Recognition

Wentao Yu, Steffen Zeiler, Dorothea Kolossa

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(title)；multi-modal(abstract)；分类 eess.AS

Comments 5 pages

Journal ref Published in Proceedings of the 28th European Signal Processing Conference (EUSIPCO), 2020

Multi-modal Egocentric Activity Recognition using Audio-Visual Features

Mehmet Ali Arabacı, Fatih Özkan, Elif Surer, Peter Jančovič, Alptekin Temizel

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

Journal ref Multimedia Tools and Applications (2020)

Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA

Donghuo Zeng, Yi Yu, Keizo Oyama

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM

Comments 8 pages, 9 figures. Accepted by ISM 2018

Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks

Jen-Cheng Hou, Syu-Siang Wang, Ying-Hui Lai, Yu Tsao, Hsiu-Wen Chang, Hsin-Min Wang

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM

Comments To appear in IEEE Transactions on Emerging Topics in Computational Intelligence. Some audio samples can be reached in this link: https://sites.google.com/view/avse2017

Deep Multimodal Learning for Audio-Visual Speech Recognition

Youssef Mroueh, Etienne Marcheret, Vaibhava Goel

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CL

Comments ICASSP 2015

Audio-Visual Segmentation via Depth-Guided Collaborative Modeling

基于深度引导协同建模的视听分割

Zhaojin Fu, Yuyang Hong, Qi Yang, Zili Wang, Kun Ding, Shiming Xiang, Bin Fan

机构 * School of Intelligent Science and Technology, University of Science and Technology Beijing（北京科技大学智能科学与技术学院） ； Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

AI总结 该研究针对现有视听分割方法未建模几何线索的问题，提出三模态框架DGCM-AVS，通过深度感知动态调制器与深度引导渐进融合模块优化，在AVSS数据集上实现M_J、M_F指标的显著提升。

Journal ref IEEE Transactions on Multimedia, 2026

LongAV-Compass: Towards Unified Evaluation of Minute-Scale Audio-Visual Generation Across T2AV, I2AV, and V2AV

LongAV-Compass：面向分钟级音视频生成在T2AV、I2AV和V2AV上的统一评估

Tengfei Liu, Yang Shi, Xuanyu Zhu, Jiafu Tang, Liu Yang, Qixun Wang, Zhuoran Zhang, Yuqi Tang, Fengxiang Wang, Yuhao Dong, Xinlong Chen, Bozhou Li, Bohan Zeng, Yue Ding, Xiaohan Zhang, Jialu Chen, Haotian Wang, Yuanxing Zhang, Pengfei Wan, Leye Wang

机构 * Peking University（北京大学） ； Kling Team（Kling团队） ； Nanjing University（南京大学） ； SJTU（上海交通大学） ； HKUST(GZ)（香港科技大学（广州）） ； Shanghai AI Lab（上海人工智能实验室） ； Nanyang Technological University（南洋理工大学） ； CASIA（中国科学院自动化所） ； Tsinghua University（清华大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；MLLM(abstract,abstract_cn)；multimodal(abstract)；分类 cs.CV、cs.MM

AI总结 针对现有评估协议局限于短片段的问题，提出LongAV-Compass基准，通过284个测试用例和统一评估框架，系统评估分钟级音视频生成在文本、图像、视频条件下的质量、一致性和对齐。

Beyond the Cartesian Illusion: Testing Two-Stage Multi-Modal Theory of Mind under Perceptual Bottlenecks

超越笛卡尔错觉：在感知瓶颈下测试双阶段多模态理论 of Mind

Yajing Zhou, Xiangyu Kong

机构 * College of Computer Science, Beijing Information Science and Technology University（北京信息科技大学计算机学院） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；MLLM(abstract,abstract_cn)；audio-visual(abstract)；分类 cs.CV、cs.AI

AI总结 本文研究了多模态大语言模型在感知瓶颈下的双阶段空间推理能力，提出了一种基于锚点的具身体验空间分解链式推理方法，以解决空间对称性和视角模糊性问题，从而提升多模态理论 of Mind 的表现。

Comments 17 pages, 3 figures

PMMTalk: Speech-Driven 3D Facial Animation from Complementary Pseudo Multi-modal Features

Tianshun Han, Shengnan Gui, Yiqing Huang, Baihui Li, Lijian Liu, Benjia Zhou, Ning Jiang, Quan Lu, Ruicong Zhi, Yanyan Liang, Du Zhang, Jun Wan

专题命中 音频语音多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；image-text(abstract)；audio-visual(abstract)

Asymmetric Hierarchical Anchoring for Robust Audio-Visual Cross-Modal Generalization

非对称层次锚定用于音频-视觉联合表示：解决信息分配模糊性以实现鲁棒跨模态泛化

Bixing Wu, Yuhong Zhao, Zongli Ye, Jiachen Lian, Xiangyu Yue, Gopala Anumanchipalli

机构 * Zhejiang University, China（浙江大学） ； University of California, Berkeley, USA（加州大学伯克利分校） ； MMLab, Chinese University of Hong Kong, China（香港中文大学MMLab） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)

AI总结 本文提出非对称层次锚定框架，通过结构化语义锚点解决跨模态泛化中的信息分配模糊性，提升语义一致性和表示纯度。

AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment

Yuqin Cao, Xiongkuo Min, Yixuan Gao, Wei Sun, Guangtao Zhai

机构 * Institute of Image Communication ； Network Engineering, Shanghai Key Laboratory of Digital Media Processing ； Transmissions, Shanghai Jiao Tong University, Shanghai ； School of Communication \& Electronic Engineering, East China Normal University, Shanghai 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(title)；分类 cs.CV、cs.MM、eess.AS

Multimodal Trustworthy Semantic Communication for Audio-Visual Event Localization

Yuandi Li, Zhe Xiang, Fei Yu, Zhangshuang Guan, Hui Ji, Zhiguo Wan, Cheng Feng

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)

Dual-Path Cross-Modal Attention for better Audio-Visual Speech Extraction

Zhongweiyang Xu, Xulin Fan, Mark Hasegawa-Johnson

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(title)；分类 cs.CV、cs.MM、eess.AS

Comments Paper Accepted by ICASSP2023

Towards Multimodal Understanding of Passenger-Vehicle Interactions in Autonomous Vehicles: Intent/Slot Recognition Utilizing Audio-Visual Data

Eda Okur, Shachi H Kumar, Saurav Sahay, Lama Nachman

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title)；分类 cs.CV、cs.CL、cs.MM

Comments Presented as a short-paper at the 23rd Workshop on the Semantics and Pragmatics of Dialogue (SemDial 2019 - LondonLogue), Sep 4-6, 2019, London, UK

Journal ref Proceedings of the 23rd Workshop on the Semantics and Pragmatics of Dialogue (SEMDIAL), pp. 213-215, London, United Kingdom, September 2019

Multimodal Transfer Deep Learning with Applications in Audio-Visual Recognition

Seungwhan Moon, Suyoun Kim, Haohan Wang

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)

Comments 6 pages, MMML workshop at NIPS 2015

OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimization

OmniFysics：通过多模态信号处理和网络优化实现物理智能进化

Minghao Han, Dingkang Yang, Yue Jiang, Yizhou Liu, Lihua Zhang

机构 * College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院） ； Fysics AI 纠错

专题命中 音频语音多模态 ：omni-modal(title,abstract)；multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)

AI总结 OmniFysics通过统一图像、音频、视频和文本的多模态信号处理，结合动态物理数据引擎和物理知识注入，实现网络化AI系统的自主优化，提升物理理解能力。

Comments This work has been submitted to the IEEE for possible publication