多模态大模型

Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation

Weiting Tan, Jiachen Lian, Hirofumi Inaguma, Paden Tomasello, Philipp Koehn, Xutai Ma

机构 * Johns Hopkins University（约翰霍普金斯大学） ； Meta AI Research（Meta AI 研究） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments EMNLP 2025 (Findings)

Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentation

Kaining Ying, Henghui Ding, Guangquan Jie, Yu-Gang Jiang

机构 * Fudan University, China（复旦大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；MLLM(abstract)；分类 cs.CV

Comments ICCV 2025, Project Page: https://henghuiding.com/OmniAVS/

HOLA: Enhancing Audio-visual Deepfake Detection via Hierarchical Contextual Aggregations and Efficient Pre-training

Xuecheng Wu, Danlei Huang, Heli Sun, Xinyi Yin, Yifan Wang, Hao Wang, Jia Zhang, Fei Wang, Peihao Guo, Suyu Xing, Junxiao Xue, Liang He

机构 * Xi'an Jiaotong University（西安交通大学） ； Zhengzhou University（郑州大学） ； University of Science and Technology of China（中国科学技术大学） ； Dalian University of Technology（大连理工大学） ； Zhejiang Lab（浙江实验室） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Multi-Microphone and Multi-Modal Emotion Recognition in Reverberant Environment

Ohad Cohen, Gershon Hazan, Sharon Gannot

机构 * Faculty of Engineering Bar-Ilan University Ramat-Gan, Israel（工程学院 巴伊兰大学 拉马特-甘, 以色列） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；multimodal(abstract)；audio-visual(abstract)；分类 eess.AS

Comments 5 pages, 4 figures, 2 tables. Accepted to EUSIPCO 2025

AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection

Ammarah Hashmi, Sahibzada Adil Shahzad, Chia-Wen Lin, Yu Tsao, Hsin-Min Wang

机构 * Social Networks and Human-Centered Computing Program, Taiwan International Graduate Program, Academia Sinica（社会网络与以人为本计算计划、台湾国际研究生计划、中国科学院） ； Institute of Information Systems and Applications, National Tsing Hua University（信息系统与应用研究所、国立清华大学） ； Department of Computer Science, National Chengchi University（计算机科学系、国立成功大学） ； Research Center for Information Technology Innovation, Academia Sinica（信息技术创新研究中心、中国科学院） ； Institute of Information Science, Academia Sinica（信息科学研究所、中国科学院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing

Mingfei Chen, Zijun Cui, Xiulong Liu, Jinlin Xiang, Caleb Zheng, Jingyuan Li, Eli Shlizerman

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Project website with demo videos: https://zijuncui02.github.io/SAVVY/

Ola: Pushing the Frontiers of Omni-Modal Language Model

Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, Yongming Rao

机构 * Tsinghua University（清华大学） ； Tencent Hunyuan Research（腾讯文睿实验室） ； S-Lab, NTU（国立科技大学S实验室） 纠错

专题命中 音频语音多模态 ：omni-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection

Peng Wu, Wanshun Su, Guansong Pang, Yujia Sun, Qingsen Yan, Peng Wang, Yanning Zhang

机构 * School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机科学学院） ； School of Computing and Information Systems, Singapore Management University（新加坡管理学院计算与信息系统学院） ； School of Artifical Intelligence, Xidian University（西安电子科技大学人工智能学院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments 12 pages, 6 figures, 9 tables. This work has been submitted to the IEEE for possible publication

Text-Audio-Visual-conditioned Diffusion Model for Video Saliency Prediction

Li Yu, Xuanzhe Sun, Wei Zhou, Moncef Gabbouj

机构 * School of Computer Science, Nanjing University of Information Science and Technology（信息科学与技术南京大学计算机科学学院） ； Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology, Nanjing University of Information Science and Technology（大气环境与设备技术江苏省协同创新中心） ； School of Computer Science and Informatics, Cardiff University（计算机科学与信息学院卡斯尔大学） ； Faculty of Information Technology and Communication Sciences, Tampere University（信息科技与通讯科学学院坦佩雷大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；image-text(abstract)；分类 cs.CV

DTFSal: Audio-Visual Dynamic Token Fusion for Video Saliency Prediction

Kiana Hooshanfar, Alireza Hosseini, Ahmad Kalhor, Babak Nadjar Araabi

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges

Ping Liu, Qiqi Tao, Joey Tianyi Zhou

专题命中 音频语音多模态 ：multi-modal(title,abstract)；multimodal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments P. Liu is with the Department of Computer Science and Engineering, University of Nevada, Reno, NV, 89512. Q. Tao and J. Zhou are with Centre for Frontier AI Research (CFAR), and Institute of High Performance Computing (IHPC), A*STAR, Singapore. J. Zhou is also with Centre for Advanced Technologies in Online Safety (CATOS), A*STAR, Singapore. J. Zhou is the corresponding author

SViQA: A Unified Speech-Vision Multimodal Model for Textless Visual Question Answering

Bingxin Li

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Continual Cross-Modal Generalization

Yan Xia, Hai Huang, Minghui Fang, Zhou Zhao

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multimodal(abstract)；image-text(abstract)；分类 cs.CV

Audio-Visual Representation Learning via Knowledge Distillation from Speech Foundation Models

Jing-Xuan Zhang, Genshun Wan, Jianqing Gao, Zhen-Hua Ling

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments accepted to Pattern Recognition

Metric Learning with Progressive Self-Distillation for Audio-Visual Embedding Learning

Donghuo Zeng, Kazushi Ikeda

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 5 pages, 3 figures, 2 tables. Accepted by ICASSP 2025

PAL: Prompting Analytic Learning with Missing Modality for Multi-Modal Class-Incremental Learning

Xianghu Yue, Yiming Chen, Xueyi Zhang, Xiaoxue Gao, Mengling Feng, Mingrui Lao, Huiping Zhuang, Haizhou Li

专题命中 音频语音多模态 ：multi-modal(title,abstract)；image-text(abstract)；audio-visual(abstract)；分类 cs.MM

WavFusion: Towards wav2vec 2.0 Multimodal Speech Emotion Recognition

Feng Li, Jiusong Luo, Wanjun Xia

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by 31st International Conference on MultiMedia Modeling (MMM2025)

Aligning Audio-Visual Joint Representations with an Agentic Workflow

Shentong Mo, Yibing Song

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event Localization

Xiang He, Xiangxi Liu, Yang Li, Dongcheng Zhao, Guobin Shen, Qingqun Kong, Xin Yang, Yi Zeng

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by ACM MM 2024. Code is available at this https://github.com/Brain-Cog-Lab/CACE-Net

Exploring Audio-Visual Information Fusion for Sound Event Localization and Detection In Low-Resource Realistic Scenarios

Ya Jiang, Qing Wang, Jun Du, Maocheng Hu, Pengfei Hu, Zeyan Liu, Shi Cheng, Zhaoxu Nian, Yuxuan Dong, Mingqi Cai, Xin Fang, Chin-Hui Lee

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments accepted by icme2024

DubWise: Video-Guided Speech Duration Control in Multimodal LLM-based Text-to-Speech for Dubbing

Neha Sahipjohn, Ashishkumar Gudmalwar, Nirmesh Shah, Pankaj Wasnik, Rajiv Ratn Shah

专题命中 音频语音多模态 ：multimodal(title)；multi-modal(abstract)；cross-modal(abstract)；audio-visual(abstract)

Comments Accepted at INTERSPEECH 2024

AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation

Jeongsoo Choi, Se Jin Park, Minsu Kim, Yong Man Ro

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments CVPR 2024. Code & Demo: https://choijeongsoo.github.io/av2av

AVT2-DWF: Improving Deepfake Detection with Audio-Visual Fusion and Dynamic Weighting Strategies

Rui Wang, Dengpan Ye, Long Tang, Yunming Zhang, Jiacheng Deng

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV

Answering Diverse Questions via Text Attached with Key Audio-Visual Clues

Qilang Ye, Zitong Yu, Xin Liu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarios

Yuanyuan Jiang, Jianqin Yin

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted to EMNLP 2023 Findings

Sound Source Localization is All about Cross-Modal Alignment

Arda Senocak, Hyeonggon Ryu, Junsik Kim, Tae-Hyun Oh, Hanspeter Pfister, Joon Son Chung

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments ICCV 2023

Text-to-feature diffusion for audio-visual few-shot learning

Otniel-Bogdan Mercea, Thomas Hummel, A. Sophia Koepke, Zeynep Akata

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments DAGM GCPR 2023

Rethinking Audio-visual Synchronization for Active Speaker Detection

Abudukelimu Wuerkaixi, You Zhang, Zhiyao Duan, Changshui Zhang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2022)

TriBERT: Full-body Human-centric Audio-visual Representation Learning for Visual Sound Separation

Tanzila Rahman, Mengyu Yang, Leonid Sigal

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments 10 pages, 5 Figures, Neurips 2021

Journal ref https://nips.cc/Conferences/2021

AVLnet: Learning Audio-Visual Language Representations from Instructional Videos

Andrew Rouditchenko, Angie Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuehne, Rameswar Panda, Rogerio Feris, Brian Kingsbury, Michael Picheny, Antonio Torralba, James Glass

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments A version of this work has been accepted to Interspeech 2021