多模态大模型

Positive Sample Propagation along the Audio-Visual Event Line

Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, Meng Wang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to CVPR 2021. Code is available at https://github.com/jasongief/PSP_CVPR_2021

Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge

Riza Velioglu, Jewgeni Rose

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Presented at NeurIPS (2020)

Jointly Optimizing Sensing Pipelines for Multimodal Mixed Reality Interaction

Darshana Rathnayake, Ashen de Silva, Dasun Puwakdandawa, Lakmal Meegahapola, Archan Misra, Indika Perera

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)

Comments 17th IEEE International Conference on Mobile Ad-Hoc and Sensor Systems (MASS) - 2020

MAST: Multimodal Abstractive Summarization with Trimodal Hierarchical Attention

Aman Khullar, Udit Arora

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments To appear in the first EMNLP Workshop on NLP Beyond Text, 2020. Aman Khullar and Udit Arora have equal contribution

Unsupervised vs. transfer learning for multimodal one-shot matching of speech and images

Leanne Nortje, Herman Kamper

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Accepted at Interspeech 2020

Strong and Simple Baselines for Multimodal Utterance Embeddings

Paul Pu Liang, Yao Chong Lim, Yao-Hung Hubert Tsai, Ruslan Salakhutdinov, Louis-Philippe Morency

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments NAACL 2019 oral presentation

deepsing: Generating Sentiment-aware Visual Stories using Cross-modal Music Translation

Nikolaos Passalis, Stavros Doropoulos

专题命中 音频语音多模态 ：cross-modal(title,abstract)；分类 cs.CV、cs.AI、eess.AS

Advances in Online Audio-Visual Meeting Transcription

Takuya Yoshioka, Igor Abramovski, Cem Aksoylar, Zhuo Chen, Moshe David, Dimitrios Dimitriadis, Yifan Gong, Ilya Gurvich, Xuedong Huang, Yan Huang, Aviv Hurvitz, Li Jiang, Sharon Koubi, Eyal Krupka, Ido Leichter, Changliang Liu, Partha Parthasarathy, Alon Vinnikov, Lingfeng Wu, Xiong Xiao, Wayne Xiong, Huaming Wang, Zhenghao Wang, Jun Zhang, Yong Zhao, Tianyan Zhou

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments To appear in Proc. IEEE ASRU Workshop 2019

Recurrent Neural Network Transducer for Audio-Visual Speech Recognition

Takaki Makino, Hank Liao, Yannis Assael, Brendan Shillingford, Basilio Garcia, Otavio Braga, Olivier Siohan

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Will be presented in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU 2019)

DurIAN: Duration Informed Attention Network For Multimodal Synthesis

Chengzhu Yu, Heng Lu, Na Hu, Meng Yu, Chao Weng, Kun Xu, Peng Liu, Deyi Tuo, Shiyin Kang, Guangzhi Lei, Dan Su, Dong Yu

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Towards Generating Ambisonics Using Audio-Visual Cue for Virtual Reality

Aakanksha Rana, Cagri Ozcinar, Aljoscha Smolic

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Speaker-Targeted Audio-Visual Models for Speech Recognition in Cocktail-Party Environments

Guan-Lin Chao, William Chan, Ian Lane

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Published in INTERSPEECH 2016

AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection

Joseph Roth, Sourish Chaudhuri, Ondrej Klejch, Radhika Marvin, Andrew Gallagher, Liat Kaver, Sharadh Ramaswamy, Arkadiusz Stopczynski, Cordelia Schmid, Zhonghua Xi, Caroline Pantofaru

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Deep Multimodal Clustering for Unsupervised Audiovisual Learning

Di Hu, Feiping Nie, Xuelong Li

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by CVPR2019

End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features

Chiori Hori, Huda Alamri, Jue Wang, Gordon Wichern, Takaaki Hori, Anoop Cherian, Tim K. Marks, Vincent Cartillier, Raphael Gontijo Lopes, Abhishek Das, Irfan Essa, Dhruv Batra, Devi Parikh

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments A prototype system for the Audio Visual Scene-aware Dialog (AVSD) at DSTC7

A Deep Learning Approach for Multimodal Deception Detection

Gangeshwar Krishnamurthy, Navonil Majumder, Soujanya Poria, Erik Cambria

专题命中 音频语音多模态 ：multimodal(title)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at the 19th International Conference on Computational Linguistics and Intelligent Text Processing (CICLing), 2018

AUREXA-SE: Audio-Visual Unified Representation Exchange Architecture with Cross-Attention and Squeezeformer for Speech Enhancement

M. Sajid, Deepanshu Gupta, Yash Modi, Sanskriti Jain, Harshith Jai Surya Ganji, A. Rahaman, Harshvardhan Choudhary, Nasir Saleem, Amir Hussain, M. Tanveer

机构 * Indian Institute of Technology Indore（印度理工学院印第安纳分校） ； School of Computing, Edinburgh Napier University（爱丁堡纳皮尔大学计算机学院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.AI、cs.MM

Journal ref INTERSPEECH 2025 - 4th COG-MHEAR Workshop on Audio-Visual Speech Enhancement (AVSEC)

A Multi-Task, Multi-Modal Approach for Predicting Categorical and Dimensional Emotions

Alex-Răzvan Ispas, Théo Deschamps-Berger, Laurence Devillers

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI；multimodal(comments)

Comments Companion Publication of the 25th International Conference on Multimodal Interaction (pp. 311-317)

Multimodal Approach for Assessing Neuromotor Coordination in Schizophrenia Using Convolutional Neural Networks

Yashish M. Siriwardena, Chris Kitchen, Deanna L. Kelly, Carol Espy-Wilson

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.MM、eess.AS

Comments 5 pages. arXiv admin note: text overlap with arXiv:2102.07054

Journal ref Proceedings of the 2021 International Conference on Multimodal Interaction

Learning Audio-Visual Dereverberation

Changan Chen, Wei Sun, David Harwath, Kristen Grauman

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted at ICASSP 2023. This is the longer version of the five-page camera-ready paper. Project page: https://vision.cs.utexas.edu/projects/learning-audio-visual-dereverberation

Shapes of Emotions: Multimodal Emotion Recognition in Conversations via Emotion Shifts

Harsh Agarwal, Keshav Bansal, Abhinav Joshi, Ashutosh Modi

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 13 pages, Accepted at Workshop on Performance and Interpretability Evaluations of Multimodal, Multipurpose, Massive-Scale Models, COLING 2022

Sound Adversarial Audio-Visual Navigation

Yinfeng Yu, Wenbing Huang, Fuchun Sun, Changan Chen, Yikai Wang, Xiaohong Liu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments This work aims to do an adversarial sound intervention for robust audio-visual navigation

Multimodal Emotion Recognition with High-level Speech and Text Features

Mariana Rodrigues Makiuchi, Kuniaki Uto, Koichi Shinoda

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted at ASRU 2021. Code available at https://github.com/mmakiuchi/multimodal_emotion_recognition

Dyadic Speech-based Affect Recognition using DAMI-P2C Parent-child Multimodal Interaction Dataset

Huili Chen, Yue Zhang, Felix Weninger, Rosalind Picard, Cynthia Breazeal, Hae Won Park

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted by the 2020 International Conference on Multimodal Interaction (ICMI'20)

Getting the subtext without the text: Scalable multimodal sentiment classification from visual and acoustic modalities

Nathaniel Blanchard, Daniel Moreira, Aparna Bharati, Walter J. Scheirer

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Published in the First Workshop on Computational Modeling of Human Multimodal Language - ACL 2018

StructBreak: Structural Cognitive Overload-Induced Safety Failures in MLLMs

StructBreak: 多模态大语言模型中结构性认知过载引发的安全故障

Yang Luo, Xinran Liu, Tiantian Ji, Zhiyi Yin, Lingyun Peng, Shuyu Li

机构 * Key Laboratory of Trustworthy Distributed Computing and Service (MoE), Beijing University of Posts and Telecommunications（可信分布式计算与服务重点实验室（MoE），北京邮电大学） ； Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所） 纠错

专题命中 音频语音多模态 ：MLLM(summary_cn,abstract_cn)；multimodal(abstract)；分类 cs.AI

AI总结 提出StructBreak框架，通过量化结构性认知过载（SCO）揭示一种高阶认知过载攻击范式，在六种主流MLLM上实现92%平均攻击成功率，并证明该攻击通过结构性通道绕过安全过滤器。

Comments 23 pages; accepted to Findings of ACL 2026. This paper contains examples of harmful content

MGM-Omni: Scaling Omni LLMs to Personalized Long-Horizon Speech

Chengyao Wang, Zhisheng Zhong, Bohao Peng, Senqiao Yang, Yuqi Liu, Haokun Gui, Bin Xia, Jingyao Li, Bei Yu, Jiaya Jia

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；omni-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Code is available at https://github.com/dvlab-research/MGM-Omni

Foundations of Multisensory Artificial Intelligence

Paul Pu Liang

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；multimodal foundation model(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CMU Machine Learning Department PhD Thesis

Can CLIP Help Sound Source Localization?

Sooyoung Park, Arda Senocak, Joon Son Chung

专题命中 音频语音多模态 ：multimodal(abstract)；image-text(abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments WACV 2024

VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning

Qiushi Zhu, Long Zhou, Ziqiang Zhang, Shujie Liu, Binxing Jiao, Jie Zhang, Lirong Dai, Daxin Jiang, Jinyu Li, Furu Wei

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 11 pages, Accepted by IEEE Transactions on Multimedia