多模态大模型 - arXivDaily 专题

1904.08248 2019-11-28 eess.AS cs.CL cs.SD stat.ML 81%

An Analysis of Speech Enhancement and Recognition Losses in Limited Resources Multi-talker Single Channel Audio-Visual ASR

Luca Pasa, Giovanni Morrone, Leonardo Badino

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CL、eess.AS

1901.02839 2019-11-20 cs.HC cs.AI cs.CV 81%

SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild

Jean Kossaifi, Robert Walecki, Yannis Panagakis, Jie Shen, Maximilian Schmitt, Fabien Ringeval, Jing Han, Vedhas Pandit, Antoine Toisoul, Bjorn Schuller, Kam Star, Elnar Hajiyev, Maja Pantic

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.AI

Journal ref IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019

1809.10961 2019-10-30 cs.CV cs.MM stat.ML 81%

Variational Bayesian Inference for Audio-Visual Tracking of Multiple Speakers

Yutong Ban, Xavier Alameda-Pineda, Laurent Girin, Radu Horaud

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM

1910.03814 2019-10-10 cs.CV cs.CL 81%

Exploring Hate Speech Detection in Multimodal Publications

Raul Gomez, Jaume Gibert, Lluis Gomez, Dimosthenis Karatzas

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL

1910.00330 2019-10-02 cs.LG cs.CL eess.AS stat.ML 81%

A Multi-Modal Feature Embedding Approach to Diagnose Alzheimer Disease from Spoken Language

S. Soroush Haj Zargarbashi, Bagher Babaali

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CL、eess.AS

Comments 14 pages, 4 figures

1909.10407 2019-09-24 cs.SD cs.CV cs.LG eess.AS 81%

CochleaNet: A Robust Language-independent Audio-Visual Model for Speech Enhancement

Mandar Gogate, Kia Dashtipour, Ahsan Adeel, Amir Hussain

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments 34 pages, 11 figures, Submitted to Information Fusion

1907.04975 2019-07-12 cs.CV cs.SD eess.AS 81%

My lips are concealed: Audio-visual speech enhancement through obstructions

Triantafyllos Afouras, Joon Son Chung, Andrew Zisserman

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted to Interspeech 2019

1907.04926 2019-07-12 eess.AS cs.MM cs.SD eess.IV 81%

Synchronizing Audio-Visual Film Stimuli in Unity (version 5.5.1f1): Game Engines as a Tool for Research

Javier Sanz, Andreas Wulff-Abramsson, Carlos Aguilar-Paredes, Luis Emilio Bruni, Lydia Sanchez

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.MM、eess.AS

Comments 13 Pages

1906.02112 2019-07-10 eess.AS cs.CV eess.IV 81%

Investigating the Lombard Effect Influence on End-to-End Audio-Visual Speech Recognition

Pingchuan Ma, Stavros Petridis, Maja Pantic

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted for publication at Interspeech 2019

1906.10042 2019-06-25 cs.SD cs.CV eess.AS 81%

Who said that?: Audio-visual speaker diarisation of real-world meetings

Joon Son Chung, Bong-Jin Lee, Icksang Han

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted to Interspeech 2019

1904.09013 2019-04-22 cs.CV cs.SD eess.AS eess.IV 81%

Self-Supervised Audio-Visual Co-Segmentation

Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh McDermott, Antonio Torralba

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted to ICASSP 2019

1805.06606 2019-03-11 cs.CL cs.AI cs.HC 81%

Convolutional Attention Networks for Multimodal Emotion Recognition from Speech and Text Data

Chan Woo Lee, Kyu Ye Song, Jihoon Jeong, Woo Yong Choi

专题命中音频语音多模态：multimodal(title)；multi-modal(abstract)；分类 cs.CL、cs.AI

Comments Inaccurate scientific facts listed in document

1902.05347 2019-02-15 cs.MM cs.IR cs.SD eess.AS 81%

Multimodal music information processing and retrieval: survey and future challenges

Federico Simonetta, Stavros Ntalampiras, Federico Avanzini

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.MM、eess.AS

Journal ref in 2019 International Workshop on Multilayer Music Representation and Processing, Milano, IEEE 2019

1812.06071 2018-12-17 cs.CV cs.LG cs.MM 81%

On Attention Modules for Audio-Visual Synchronization

Naji Khosravan, Shervin Ardeshir, Rohit Puri

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM

1811.10813 2018-11-28 cs.CV eess.AS 81%

Noise-tolerant Audio-visual Online Person Verification using an Attention-based Neural Network Fusion

Suwon Shon, Tae-Hyun Oh, James Glass

专题命中音频语音多模态：audio-visual(title)；multi-modal(abstract)；分类 cs.CV、eess.AS

1804.03641 2018-10-10 cs.CV cs.SD eess.AS 81%

Audio-Visual Scene Analysis with Self-Supervised Multisensory Features

Andrew Owens, Alexei A. Efros

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、eess.AS

1809.00758 2018-10-03 cs.LG cs.CV cs.SD eess.AS stat.ML 81%

End-to-end Multimodal Emotion and Gender Recognition with Dynamic Joint Loss Weights

Myungsu Chae, Tae-Ho Kim, Young Hoon Shin, June-Woo Kim, Soo-Young Lee

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、eess.AS

Comments IROS 2018 Workshop on Crossmodal Learning for Intelligent Robotics

1805.07467 2018-09-24 cs.CL cs.SD eess.AS 81%

Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces

Yu-An Chung, Wei-Hung Weng, Schrasing Tong, James Glass

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted to NIPS 2018. v2 added the majority word baseline results and other minor fixes. arXiv admin note: text overlap with arXiv:1710.04087 by other authors

1804.03619 2018-08-13 cs.SD cs.CV eess.AS 81%

Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation

Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T. Freeman, Michael Rubinstein

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted to SIGGRAPH 2018. Project webpage: https://looking-to-listen.github.io

Journal ref ACM Trans. Graph. 37(4): 112:1-112:11 (2018)

1803.10299 2018-06-19 cs.CL cs.SD eess.AS 81%

Multi-Modal Data Augmentation for End-to-End ASR

Adithya Renduchintala, Shuoyang Ding, Matthew Wiesner, Shinji Watanabe

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CL、eess.AS

Comments 5 Pages, 1 Figure, accepted at INTERSPEECH 2018

1802.05521 2018-02-16 cs.CV cs.SD eess.AS 81%

Deep Learning for Lip Reading using Audio-Visual Information for Urdu Language

M Faisal, Sanaullah Manzoor

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、eess.AS

1511.04401 2017-12-08 cs.CV cs.CL cs.LG cs.NE 81%

Symbol Grounding Association in Multimodal Sequences with Missing Elements

Federico Raue, Andreas Dengel, Thomas M. Breuel, Marcus Liwicki

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Under review on Journal of Artificial Intelligence Research (JAIR) -- Special Track on Deep Learning, Knowledge Representation, and Reasoning

1707.09538 2017-08-01 cs.MM cs.CL 81%

Benchmarking Multimodal Sentiment Analysis

Erik Cambria, Devamanyu Hazarika, Soujanya Poria, Amir Hussain, R. B. V. Subramaanyam

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL、cs.MM

Comments Accepted in CICLing 2017

1703.01725 2017-03-07 cs.SI cs.CL cs.CV physics.soc-ph 81%

Cats and Captions vs. Creators and the Clock: Comparing Multimodal Content to Context in Predicting Relative Popularity

Jack Hessel, Lillian Lee, David Mimno

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments 10 pages, data and models available at http://www.cs.cornell.edu/~jhessel/cats/cats.html, Proceedings of WWW 2017

1702.06151 2017-02-22 cs.CV cs.IR cs.LG cs.MM 81%

Developing a comprehensive framework for multimodal feature extraction

Quinten McNamara, Alejandro de la Vega, Tal Yarkoni

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.MM

1606.06259 2016-11-18 cs.CL cs.MM 81%

MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos

Amir Zadeh, Rowan Zellers, Eli Pincus, Louis-Philippe Morency

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL、cs.MM

Comments Accepted as Journal Publication in IEEE Intelligent Systems

Journal ref IEEE Intelligent Systems 31.6 (2016): 82-88

cs/0505064 2016-11-17 cs.HC cs.AI cs.CV cs.LG cs.RO 81%

Multi-Modal Human-Machine Communication for Instructing Robot Grasping Tasks

P. C. McGuire, J. Fritsch, J. J. Steil, F. Roethling, G. A. Fink, S. Wachsmuth, G. Sagerer, H. Ritter

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments 7 pages, 8 figures

Journal ref Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Lausanne, Switzerland, IEEE publications, pp. 1082-1089 (2002)

2608.23759 2026-08-26 eess.AS cs.SD 新提交 80%

The ISCSLP 2026 Real-World Audio-Visual Speech Enhancement Challenge

ISCSLP 2026 真实场景视听语音增强挑战赛

Challenge Organizers

专题命中音频语音多模态：audio-visual(title,abstract)；分类 eess.AS

AI总结该研究介绍ISCSLP 2026真实场景视听语音增强挑战赛，设置含真实混合、合成重混等赛道，公布基线结果与相关资源，以评估视听语音增强在自然场景下的性能。

Comments The First Real-World Audio-Visual Speech Enhancement (AVSE) Challenge

2608.18401 2026-08-20 cs.HC cs.CL cs.RO 新提交 80%

Multimodal Rapport Estimation in Real-World HRI

现实世界人机交互（HRI）中的多模态融洽度估计

Akihiro Sakuramoto, Takato Hayashi, Ryo Miyoshi, Yuki Okafuji, Shogo Okada

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL

AI总结本研究针对现实世界人机交互的融洽度估计问题，采用日本药店的62个多模态会话，对比零样本LLMs等模型，发现Gemini融合模型表现最优，且性能随互动时长、群体规模变化。

Comments 9 pages, 4 figures, 3 tables. Accepted at the 28th ACM International Conference on Multimodal Interaction (ICMI 2026)

2508.00265 2025-08-06 cs.CV 80%

Multimodal Referring Segmentation: A Survey

Henghui Ding, Song Tang, Shuting He, Chang Liu, Zuxuan Wu, Yu-Gang Jiang

机构 * Fudan University（复旦大学）； Shanghai University of Finance and Economics（上海财经大学）； ByteDance Inc（字节跳动公司）

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV

Comments Project Page: https://github.com/henghuiding/Awesome-Multimodal-Referring-Segmentation

1. 音频语音多模态 4585 篇

An Analysis of Speech Enhancement and Recognition Losses in Limited Resources Multi-talker Single Channel Audio-Visual ASR

SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild

Variational Bayesian Inference for Audio-Visual Tracking of Multiple Speakers

Exploring Hate Speech Detection in Multimodal Publications

A Multi-Modal Feature Embedding Approach to Diagnose Alzheimer Disease from Spoken Language

CochleaNet: A Robust Language-independent Audio-Visual Model for Speech Enhancement

My lips are concealed: Audio-visual speech enhancement through obstructions

Synchronizing Audio-Visual Film Stimuli in Unity (version 5.5.1f1): Game Engines as a Tool for Research

Investigating the Lombard Effect Influence on End-to-End Audio-Visual Speech Recognition

Who said that?: Audio-visual speaker diarisation of real-world meetings

Self-Supervised Audio-Visual Co-Segmentation

Convolutional Attention Networks for Multimodal Emotion Recognition from Speech and Text Data

Multimodal music information processing and retrieval: survey and future challenges

On Attention Modules for Audio-Visual Synchronization

Noise-tolerant Audio-visual Online Person Verification using an Attention-based Neural Network Fusion

Audio-Visual Scene Analysis with Self-Supervised Multisensory Features

End-to-end Multimodal Emotion and Gender Recognition with Dynamic Joint Loss Weights

Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces

Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation

Multi-Modal Data Augmentation for End-to-End ASR

Deep Learning for Lip Reading using Audio-Visual Information for Urdu Language

Symbol Grounding Association in Multimodal Sequences with Missing Elements

Benchmarking Multimodal Sentiment Analysis

Cats and Captions vs. Creators and the Clock: Comparing Multimodal Content to Context in Predicting Relative Popularity

Developing a comprehensive framework for multimodal feature extraction

MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos

Multi-Modal Human-Machine Communication for Instructing Robot Grasping Tasks

The ISCSLP 2026 Real-World Audio-Visual Speech Enhancement Challenge

Multimodal Rapport Estimation in Real-World HRI

Multimodal Referring Segmentation: A Survey