多模态大模型 - arXivDaily 专题

2505.21093 2025-05-28 eess.AS cs.SD 88%

Multimodal Assessment of Speech Impairment in ALS Using Audio-Visual and Machine Learning Approaches

Francesco Pierotti, Andrea Bandini

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Comments Submitted to Interspeech

2409.07967 2025-05-12 cs.CV 88%

Locality-aware Cross-modal Correspondence Learning for Dense Audio-Visual Events Localization

Ling Xing, Hongyu Qu, Rui Yan, Xiangbo Shu, Jinhui Tang

机构 * School of Computer Science and Engineering, Nanjing University of Science and Technology（计算机科学与工程学院，南京理工大学）

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

2503.23990 2025-04-01 cs.CL 88%

BeMERC: Behavior-Aware MLLM-based Framework for Multimodal Emotion Recognition in Conversation

Yumeng Fu, Junjie Wu, Zhongjie Wang, Meishan Zhang, Yulin Wu, Bingquan Liu

专题命中音频语音多模态：multimodal(title,abstract)；MLLM(title,abstract)；分类 cs.CL

2403.14168 2024-06-05 cs.CL 88%

M$^3$AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset

Zhe Chen, Heyang Liu, Wenyi Yu, Guangzhi Sun, Hongcheng Liu, Ji Wu, Chao Zhang, Yu Wang, Yanfeng Wang

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CL

Comments ACL 2024 Main Conference. Project website: https://jack-zc8.github.io/M3AV-dataset-page

2404.06309 2024-04-10 cs.CV 88%

Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models

David Kurzendörfer, Otniel-Bogdan Mercea, A. Sophia Koepke, Zeynep Akata

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

Comments CVPRw 2024 (L3D-IVU)

2309.08348 2023-09-18 eess.AS cs.SD 88%

The Multimodal Information Based Speech Processing (MISP) 2023 Challenge: Audio-Visual Target Speaker Extraction

Shilong Wu, Chenxi Wang, Hang Chen, Yusheng Dai, Chenyue Zhang, Ruoyu Wang, Hongbo Lan, Jun Du, Chin-Hui Lee, Jingdong Chen, Shinji Watanabe, Sabato Marco Siniscalchi, Odette Scharenborg, Zhong-Qiu Wang, Jia Pan, Jianqing Gao

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Comments 5 pages, 4 figures

2307.02041 2023-07-06 cs.CV 88%

Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised Audio-Visual Video Parsing

Jie Fu, Junyu Gao, Changsheng Xu

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

2210.17456 2023-06-02 eess.AS cs.SD 88%

Audio-Visual Speech Enhancement and Separation by Utilizing Multi-Modal Self-Supervised Embeddings

I-Chun Chern, Kuo-Hsuan Hung, Yi-Ting Chen, Tassadaq Hussain, Mandar Gogate, Amir Hussain, Yu Tsao, Jen-Cheng Hou

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Comments ICASSP AMHAT 2023

2303.06326 2023-03-14 cs.MM 88%

The Multimodal Information based Speech Processing (MISP) 2022 Challenge: Audio-Visual Diarization and Recognition

Zhe Wang, Shilong Wu, Hang Chen, Mao-Kui He, Jun Du, Chin-Hui Lee, Jingdong Chen, Shinji Watanabe, Sabato Siniscalchi, Odette Scharenborg, Diyuan Liu, Baocai Yin, Jia Pan, Jianqing Gao, Cong Liu

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(title)；multi-modal(abstract)；分类 cs.MM

Comments 5 pages, 4 figures, to be published in ICASSP2023

2212.02782 2022-12-07 eess.AS cs.SD 88%

Self-Supervised Audio-Visual Speech Representations Learning By Multimodal Self-Distillation

Jing-Xuan Zhang, Genshun Wan, Zhen-Hua Ling, Jia Pan, Jianqing Gao, Cong Liu

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Comments submitted to ICASSP 2023

2111.05329 2022-11-28 cs.CV cs.LG 88%

Self-Supervised Audio-Visual Representation Learning with Relaxed Cross-Modal Synchronicity

Pritam Sarkar, Ali Etemad

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

Comments Accepted in AAAI 2023

2210.13127 2022-10-25 eess.AS cs.SD eess.SP 88%

A Novel Frame Structure for Cloud-Based Audio-Visual Speech Enhancement in Multimodal Hearing-aids

Abhijeet Bishnu, Ankit Gupta, Mandar Gogate, Kia Dashtipour, Ahsan Adeel, Amir Hussain, Mathini Sellathurai, Tharmalingam Ratnarajah

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

2202.04528 2022-09-19 cs.SD cs.LG eess.AS 88%

Multimodal Audio-Visual Information Fusion using Canonical-Correlated Graph Neural Network for Energy-Efficient Speech Enhancement

Leandro Aparecido Passos, João Paulo Papa, Javier Del Ser, Amir Hussain, Ahsan Adeel

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

2112.07423 2021-12-15 cs.CV 88%

Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking

Yidi Li, Hong Liu, Hao Tang

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(title)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by AAAI2022

2110.13556 2021-10-27 cs.MM 88%

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Donghuo Zeng, Jianming Wu, Gen Hattori, Yi Yu, Rong Xu

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM

2108.00443 2021-08-03 eess.AS cs.SD 88%

A Survey on Audio Synthesis and Audio-Visual Multimodal Processing

Zhaofeng Shi

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

2106.11411 2021-06-23 cs.SD eess.AS 88%

Attention-based cross-modal fusion for audio-visual voice activity detection in musical video streams

Yuanbo Hou, Zhesong Yu, Xia Liang, Xingjian Du, Bilei Zhu, Zejun Ma, Dick Botteldooren

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Comments Accepted by INTERSPEECH 2021

2105.14430 2021-06-01 cs.CV 88%

Rethinking the constraints of multimodal fusion: case study in Weakly-Supervised Audio-Visual Video Parsing

Jianning Wu, Zhuqing Jiang, Shiping Wen, Aidong Men, Haiying Wang

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

2101.03149 2021-04-07 cs.CV cs.SD eess.IV 88%

VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency

Ruohan Gao, Kristen Grauman

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

Comments In CVPR 2021. Project page: http://vision.cs.utexas.edu/projects/VisualVoice/

2004.12943 2021-03-31 cs.CV 88%

Audio-Visual Instance Discrimination with Cross-Modal Agreement

Pedro Morgado, Nuno Vasconcelos, Ishan Misra

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

2103.01463 2021-03-03 cs.SD cs.LG eess.AS 88%

Audio-Visual Speech Separation Using Cross-Modal Correspondence Loss

Naoki Makishima, Mana Ihori, Akihiko Takashima, Tomohiro Tanaka, Shota Orihashi, Ryo Masumura

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Comments Accepted to ICASSP 2021

2008.03894 2020-08-11 eess.AS 88%

Audio-visual Speaker Recognition with a Cross-modal Discriminative Network

Ruijie Tao, Rohan Kumar Das, Haizhou Li

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

2007.14223 2020-07-29 eess.AS cs.SD 88%

Multimodal Integration for Large-Vocabulary Audio-Visual Speech Recognition

Wentao Yu, Steffen Zeiler, Dorothea Kolossa

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(title)；multi-modal(abstract)；分类 eess.AS

Comments 5 pages

Journal ref Published in Proceedings of the 28th European Signal Processing Conference (EUSIPCO), 2020

1807.00612 2020-05-01 cs.CV 88%

Multi-modal Egocentric Activity Recognition using Audio-Visual Features

Mehmet Ali Arabacı, Fatih Özkan, Elif Surer, Peter Jančovič, Alptekin Temizel

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

Journal ref Multimedia Tools and Applications (2020)

1908.03744 2019-08-13 cs.MM cs.IR 88%

Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA

Donghuo Zeng, Yi Yu, Keizo Oyama

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM

Comments 8 pages, 9 figures. Accepted by ISM 2018

1703.10893 2018-01-25 cs.SD cs.MM stat.ML 88%

Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks

Jen-Cheng Hou, Syu-Siang Wang, Ying-Hui Lai, Yu Tsao, Hsiu-Wen Chang, Hsin-Min Wang

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM

Comments To appear in IEEE Transactions on Emerging Topics in Computational Intelligence. Some audio samples can be reached in this link: https://sites.google.com/view/avse2017

1501.05396 2015-01-23 cs.CL cs.LG 88%

Deep Multimodal Learning for Audio-Visual Speech Recognition

Youssef Mroueh, Etienne Marcheret, Vaibhava Goel

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CL

Comments ICASSP 2015

2608.16285 2026-08-18 cs.CV cs.AI 新提交 88%

Audio-Visual Segmentation via Depth-Guided Collaborative Modeling

基于深度引导协同建模的视听分割

Zhaojin Fu, Yuyang Hong, Qi Yang, Zili Wang, Kun Ding, Shiming Xiang, Bin Fan

机构 * School of Intelligent Science and Technology, University of Science and Technology Beijing（北京科技大学智能科学与技术学院）； Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

AI总结该研究针对现有视听分割方法未建模几何线索的问题，提出三模态框架DGCM-AVS，通过深度感知动态调制器与深度引导渐进融合模块优化，在AVSS数据集上实现M_J、M_F指标的显著提升。

Journal ref IEEE Transactions on Multimedia, 2026

2605.26244 2026-05-27 cs.CV cs.MM cs.SD 88%

LongAV-Compass: Towards Unified Evaluation of Minute-Scale Audio-Visual Generation Across T2AV, I2AV, and V2AV

LongAV-Compass：面向分钟级音视频生成在T2AV、I2AV和V2AV上的统一评估

Tengfei Liu, Yang Shi, Xuanyu Zhu, Jiafu Tang, Liu Yang, Qixun Wang, Zhuoran Zhang, Yuqi Tang, Fengxiang Wang, Yuhao Dong, Xinlong Chen, Bozhou Li, Bohan Zeng, Yue Ding, Xiaohan Zhang, Jialu Chen, Haotian Wang, Yuanxing Zhang, Pengfei Wan, Leye Wang

机构 * Peking University（北京大学）； Kling Team（Kling团队）； Nanjing University（南京大学）； SJTU（上海交通大学）； HKUST(GZ)（香港科技大学（广州））； Shanghai AI Lab（上海人工智能实验室）； Nanyang Technological University（南洋理工大学）； CASIA（中国科学院自动化所）； Tsinghua University（清华大学）

专题命中音频语音多模态：audio-visual(title,abstract)；MLLM(abstract,abstract_cn)；multimodal(abstract)；分类 cs.CV、cs.MM

AI总结针对现有评估协议局限于短片段的问题，提出LongAV-Compass基准，通过284个测试用例和统一评估框架，系统评估分钟级音视频生成在文本、图像、视频条件下的质量、一致性和对齐。

2605.18194 2026-05-19 cs.AI cs.CV 88%

Beyond the Cartesian Illusion: Testing Two-Stage Multi-Modal Theory of Mind under Perceptual Bottlenecks

超越笛卡尔错觉：在感知瓶颈下测试双阶段多模态理论 of Mind

Yajing Zhou, Xiangyu Kong

机构 * College of Computer Science, Beijing Information Science and Technology University（北京信息科技大学计算机学院）

专题命中音频语音多模态：multi-modal(title,abstract)；MLLM(abstract,abstract_cn)；audio-visual(abstract)；分类 cs.CV、cs.AI

AI总结本文研究了多模态大语言模型在感知瓶颈下的双阶段空间推理能力，提出了一种基于锚点的具身体验空间分解链式推理方法，以解决空间对称性和视角模糊性问题，从而提升多模态理论 of Mind 的表现。

Comments 17 pages, 3 figures

1. 音频语音多模态 4567 篇

Multimodal Assessment of Speech Impairment in ALS Using Audio-Visual and Machine Learning Approaches

Locality-aware Cross-modal Correspondence Learning for Dense Audio-Visual Events Localization

BeMERC: Behavior-Aware MLLM-based Framework for Multimodal Emotion Recognition in Conversation

M$^3$AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset

Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models

The Multimodal Information Based Speech Processing (MISP) 2023 Challenge: Audio-Visual Target Speaker Extraction

Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised Audio-Visual Video Parsing

Audio-Visual Speech Enhancement and Separation by Utilizing Multi-Modal Self-Supervised Embeddings

The Multimodal Information based Speech Processing (MISP) 2022 Challenge: Audio-Visual Diarization and Recognition

Self-Supervised Audio-Visual Speech Representations Learning By Multimodal Self-Distillation

Self-Supervised Audio-Visual Representation Learning with Relaxed Cross-Modal Synchronicity

A Novel Frame Structure for Cloud-Based Audio-Visual Speech Enhancement in Multimodal Hearing-aids

Multimodal Audio-Visual Information Fusion using Canonical-Correlated Graph Neural Network for Energy-Efficient Speech Enhancement

Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

A Survey on Audio Synthesis and Audio-Visual Multimodal Processing

Attention-based cross-modal fusion for audio-visual voice activity detection in musical video streams

Rethinking the constraints of multimodal fusion: case study in Weakly-Supervised Audio-Visual Video Parsing

VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency

Audio-Visual Instance Discrimination with Cross-Modal Agreement

Audio-Visual Speech Separation Using Cross-Modal Correspondence Loss

Audio-visual Speaker Recognition with a Cross-modal Discriminative Network

Multimodal Integration for Large-Vocabulary Audio-Visual Speech Recognition

Multi-modal Egocentric Activity Recognition using Audio-Visual Features

Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA

Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks

Deep Multimodal Learning for Audio-Visual Speech Recognition

Audio-Visual Segmentation via Depth-Guided Collaborative Modeling

LongAV-Compass: Towards Unified Evaluation of Minute-Scale Audio-Visual Generation Across T2AV, I2AV, and V2AV

Beyond the Cartesian Illusion: Testing Two-Stage Multi-Modal Theory of Mind under Perceptual Bottlenecks