多模态大模型

Towards General Auditory Intelligence: Large Multimodal Models for Machine Listening and Speaking

Siyin Wang, Zengrui Jin, Changli Tang, Qiujia Li, Bo Li, Chen Chen, Yuchen Hu, Wenyi Yu, Yixuan Li, Jimin Zhuang, Yudong Yang, Mingqiu Wang, Michael Han, Yifan Ding, Junwen Bai, Tom Ouyang, Shuo-yiin Chang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Guangzhi Sun, Zhehuai Chen, Ji Wu, Bowen Zhou, Yuxuan Wang, Tara Sainath, Yonghui Wu, Chao Zhang

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 eess.AS

Comments 22 pages, 11 figures

Complementary and Contrastive Learning for Audio-Visual Segmentation

Sitong Gong, Yunzhi Zhuge, Lu Zhang, Pingping Zhang, Huchuan Lu

机构 * School of Information and Communication Engineering, Dalian University of Technology（信息与通信工程学院，大连理工大学） ； School of Future Technology and the School of Artificial Intelligence, Dalian University of Technology（未来技术学院和人工智能学院，大连理工大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted to IEEE Transactions on Multimedia

AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding

Chaeyoung Jung, Youngjoon Jang, Joon Son Chung

机构 * Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Scalable Audio-Visual Masked Autoencoders for Efficient Affective Video Facial Analysis

Xuecheng Wu, Junxiao Xue, Xinyi Yin, Yunyun Shi, Liangyu Fu, Danlei Huang, Yifan Wang, Jia Zhang, Jiayu Nie, Jun Wang

机构 * School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院） ； School of Cyber Science and Engineering, Zhengzhou University（郑州大学网络科学与工程学院） ； School of Software, Northwestern Polytechnical University（西北工业大学软件学院） ； Institute of Advanced Technology, University of Science and Technology of China（中国科学技术大学先进技术研究院） ； Inspur Group（Inspur集团） ； Research Center for Space Computing System, Zhejiang Lab（浙江实验室空间计算系统研究中心） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV

Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models

Leyi Pan, Zheyu Fu, Yunpeng Zhai, Shuchang Tao, Sheng Guan, Shiyu Huang, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Felix Henry, Aiwei Liu, Lijie Wen

机构 * Tsinghua University（清华大学） ； Tongyi Lab（通义实验室） ； Peking University（北京大学） ； OpenRL Lab（OpenRL实验室） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；omni-modal(abstract)；分类 cs.CL

Comments 22 pages, 10 figures, 12 tables

Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation

Yunzhe Shen, Kai Peng, Leiye Liu, Wei Ji, Jingjing Li, Miao Zhang, Yongri Piao, Huchuan Lu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Kling-Avatar: Grounding Multimodal Instructions for Cascaded Long-Duration Avatar Animation Synthesis

Yikang Ding, Jiwen Liu, Wenyuan Zhang, Zekun Wang, Wentao Hu, Liyuan Cui, Mingming Lao, Yingchao Shao, Hui Liu, Xiaohan Li, Ming Chen, Xiaoqiang Liu, Yu-Shen Liu, Pengfei Wan

机构 * Kling Team, Kuaishou Technology（快手科技 Kling 团队） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；MLLM(abstract)；audio-visual(abstract)；分类 cs.CV

Comments Technical Report. Project Page: https://klingavatar.github.io/

OLKAVS: An Open Large-Scale Korean Audio-Visual Speech Dataset

Jeongkyun Park, Jung-Wook Hwang, Kwanghee Choi, Seung-Hyun Lee, Jun Hwan Ahn, Rae-Hong Park, Hyung-Min Park

机构 * 1 Department of Artificial Intelligence, Sogang University, Seoul 04107, Republic of Korea 2 Department of Electronic Engineering, Sogang University, Seoul 04107, Republic of Korea 3 Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA 4 Mindslab Inc., Gyeonggi-do 13493, Republic of Korea 5 ICT Convergence Disaster/Safety Research Institute, Sogang University, Seoul 04107, Republic of Korea 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to ICASSP 2024

Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation

Weiting Tan, Jiachen Lian, Hirofumi Inaguma, Paden Tomasello, Philipp Koehn, Xutai Ma

机构 * Johns Hopkins University（约翰霍普金斯大学） ； Meta AI Research（Meta AI 研究） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments EMNLP 2025 (Findings)

Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentation

Kaining Ying, Henghui Ding, Guangquan Jie, Yu-Gang Jiang

机构 * Fudan University, China（复旦大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；MLLM(abstract)；分类 cs.CV

Comments ICCV 2025, Project Page: https://henghuiding.com/OmniAVS/

HOLA: Enhancing Audio-visual Deepfake Detection via Hierarchical Contextual Aggregations and Efficient Pre-training

Xuecheng Wu, Danlei Huang, Heli Sun, Xinyi Yin, Yifan Wang, Hao Wang, Jia Zhang, Fei Wang, Peihao Guo, Suyu Xing, Junxiao Xue, Liang He

机构 * Xi'an Jiaotong University（西安交通大学） ； Zhengzhou University（郑州大学） ； University of Science and Technology of China（中国科学技术大学） ； Dalian University of Technology（大连理工大学） ； Zhejiang Lab（浙江实验室） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Multi-Microphone and Multi-Modal Emotion Recognition in Reverberant Environment

Ohad Cohen, Gershon Hazan, Sharon Gannot

机构 * Faculty of Engineering Bar-Ilan University Ramat-Gan, Israel（工程学院 巴伊兰大学 拉马特-甘, 以色列） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；multimodal(abstract)；audio-visual(abstract)；分类 eess.AS

Comments 5 pages, 4 figures, 2 tables. Accepted to EUSIPCO 2025

AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection

Ammarah Hashmi, Sahibzada Adil Shahzad, Chia-Wen Lin, Yu Tsao, Hsin-Min Wang

机构 * Social Networks and Human-Centered Computing Program, Taiwan International Graduate Program, Academia Sinica（社会网络与以人为本计算计划、台湾国际研究生计划、中国科学院） ； Institute of Information Systems and Applications, National Tsing Hua University（信息系统与应用研究所、国立清华大学） ； Department of Computer Science, National Chengchi University（计算机科学系、国立成功大学） ； Research Center for Information Technology Innovation, Academia Sinica（信息技术创新研究中心、中国科学院） ； Institute of Information Science, Academia Sinica（信息科学研究所、中国科学院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing

Mingfei Chen, Zijun Cui, Xiulong Liu, Jinlin Xiang, Caleb Zheng, Jingyuan Li, Eli Shlizerman

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Project website with demo videos: https://zijuncui02.github.io/SAVVY/

Ola: Pushing the Frontiers of Omni-Modal Language Model

Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, Yongming Rao

机构 * Tsinghua University（清华大学） ； Tencent Hunyuan Research（腾讯文睿实验室） ； S-Lab, NTU（国立科技大学S实验室） 纠错

专题命中 音频语音多模态 ：omni-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection

Peng Wu, Wanshun Su, Guansong Pang, Yujia Sun, Qingsen Yan, Peng Wang, Yanning Zhang

机构 * School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机科学学院） ； School of Computing and Information Systems, Singapore Management University（新加坡管理学院计算与信息系统学院） ； School of Artifical Intelligence, Xidian University（西安电子科技大学人工智能学院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments 12 pages, 6 figures, 9 tables. This work has been submitted to the IEEE for possible publication

Text-Audio-Visual-conditioned Diffusion Model for Video Saliency Prediction

Li Yu, Xuanzhe Sun, Wei Zhou, Moncef Gabbouj

机构 * School of Computer Science, Nanjing University of Information Science and Technology（信息科学与技术南京大学计算机科学学院） ； Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology, Nanjing University of Information Science and Technology（大气环境与设备技术江苏省协同创新中心） ； School of Computer Science and Informatics, Cardiff University（计算机科学与信息学院卡斯尔大学） ； Faculty of Information Technology and Communication Sciences, Tampere University（信息科技与通讯科学学院坦佩雷大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；image-text(abstract)；分类 cs.CV

DTFSal: Audio-Visual Dynamic Token Fusion for Video Saliency Prediction

Kiana Hooshanfar, Alireza Hosseini, Ahmad Kalhor, Babak Nadjar Araabi

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges

Ping Liu, Qiqi Tao, Joey Tianyi Zhou

专题命中 音频语音多模态 ：multi-modal(title,abstract)；multimodal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments P. Liu is with the Department of Computer Science and Engineering, University of Nevada, Reno, NV, 89512. Q. Tao and J. Zhou are with Centre for Frontier AI Research (CFAR), and Institute of High Performance Computing (IHPC), A*STAR, Singapore. J. Zhou is also with Centre for Advanced Technologies in Online Safety (CATOS), A*STAR, Singapore. J. Zhou is the corresponding author

SViQA: A Unified Speech-Vision Multimodal Model for Textless Visual Question Answering

Bingxin Li

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Continual Cross-Modal Generalization

Yan Xia, Hai Huang, Minghui Fang, Zhou Zhao

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multimodal(abstract)；image-text(abstract)；分类 cs.CV

Audio-Visual Representation Learning via Knowledge Distillation from Speech Foundation Models

Jing-Xuan Zhang, Genshun Wan, Jianqing Gao, Zhen-Hua Ling

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments accepted to Pattern Recognition

Metric Learning with Progressive Self-Distillation for Audio-Visual Embedding Learning

Donghuo Zeng, Kazushi Ikeda

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 5 pages, 3 figures, 2 tables. Accepted by ICASSP 2025

PAL: Prompting Analytic Learning with Missing Modality for Multi-Modal Class-Incremental Learning

Xianghu Yue, Yiming Chen, Xueyi Zhang, Xiaoxue Gao, Mengling Feng, Mingrui Lao, Huiping Zhuang, Haizhou Li

专题命中 音频语音多模态 ：multi-modal(title,abstract)；image-text(abstract)；audio-visual(abstract)；分类 cs.MM

WavFusion: Towards wav2vec 2.0 Multimodal Speech Emotion Recognition

Feng Li, Jiusong Luo, Wanjun Xia

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by 31st International Conference on MultiMedia Modeling (MMM2025)

Aligning Audio-Visual Joint Representations with an Agentic Workflow

Shentong Mo, Yibing Song

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event Localization

Xiang He, Xiangxi Liu, Yang Li, Dongcheng Zhao, Guobin Shen, Qingqun Kong, Xin Yang, Yi Zeng

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by ACM MM 2024. Code is available at this https://github.com/Brain-Cog-Lab/CACE-Net

Exploring Audio-Visual Information Fusion for Sound Event Localization and Detection In Low-Resource Realistic Scenarios

Ya Jiang, Qing Wang, Jun Du, Maocheng Hu, Pengfei Hu, Zeyan Liu, Shi Cheng, Zhaoxu Nian, Yuxuan Dong, Mingqi Cai, Xin Fang, Chin-Hui Lee

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments accepted by icme2024

DubWise: Video-Guided Speech Duration Control in Multimodal LLM-based Text-to-Speech for Dubbing

Neha Sahipjohn, Ashishkumar Gudmalwar, Nirmesh Shah, Pankaj Wasnik, Rajiv Ratn Shah

专题命中 音频语音多模态 ：multimodal(title)；multi-modal(abstract)；cross-modal(abstract)；audio-visual(abstract)

Comments Accepted at INTERSPEECH 2024

AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation

Jeongsoo Choi, Se Jin Park, Minsu Kim, Yong Man Ro

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments CVPR 2024. Code & Demo: https://choijeongsoo.github.io/av2av