多模态大模型

SightSound-R1: Cross-Modal Reasoning Distillation from Vision to Audio Language Models

Qiaolin Wang, Xilin Jiang, Linyang He, Junkai Wu, Nima Mesgarani

机构 * Columbia University（哥伦比亚大学） ； University of Washington（华盛顿大学） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(abstract)；分类 cs.CL、cs.AI、eess.AS

Out-Of-Distribution Detection for Audio-visual Generalized Zero-Shot Learning: A General Framework

Liuyuan Wen

机构 * School of Physical Sciences University of Science and Technology of China（中国科学技术大学物理科学学院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to BMVC 2024

AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling

Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang, Xipeng Qiu

机构 * Fudan University（复旦大学） ； Multimodal Art Projection Research Community（多模态艺术投影研究社区） ； Shanghai AI Laboratory（上海人工智能实验室） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；any-to-any(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 28 pages, 16 figures, under review, work in progress

AD-AVSR: Asymmetric Dual-stream Enhancement for Robust Audio-Visual Speech Recognition

Junxiao Xue, Xiaozhen Liu, Xuecheng Wu, Xinyi Yin, Danlei Huang, Fei Yu

机构 * Zhengzhou University（郑州大学） ； Zhejiang Lab（浙江实验室） ； Xi'an Jiaotong University（西安交通大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by the ACM MM 2025 Workshop on SVC

AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generation

Le Wang, Jun Wang, Chunyu Qiang, Feng Deng, Chen Zhang, Di Zhang, Kun Gai

机构 * China University of Mining and Technology（中国矿业大学） ； Kuaishou Technology（快手科技） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 12 pages, 2 figures

Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation

Jinxing Zhou, Yanghao Zhou, Mingfei Han, Tong Wang, Xiaojun Chang, Hisham Cholakkal, Rao Muhammad Anwer

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Project page: https://github.com/jasongief/TGS-Agent

SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec

Chunyu Qiang, Haoyu Wang, Cheng Gong, Tianrui Wang, Ruibo Fu, Tao Wang, Ruilong Chen, Jiangyan Yi, Zhengqi Wen, Chen Zhang, Longbiao Wang, Jianwu Dang, Jianhua Tao

机构 * School of New Media and Communication, Tianjin University（新媒体与传播学院，天津大学） ； Kuaishou Technology（快手科技） ； Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University（认知计算与应用重点实验室，智能计算学院，天津大学） ； Department of Automation, BNRist, Tsinghua University（自动化系，北研所，清华大学） ； Shenzhen Institute of Advanced Technology, Chinese Academy of Science（深圳先进技术研究院，中国科学院） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Contrastive Conditional Latent Diffusion for Audio-visual Segmentation

Yuxin Mao, Jing Zhang, Mochu Xiang, Yunqiu Lv, Dong Li, Yiran Zhong, Yuchao Dai

机构 * School of Electronics and Information, Northwestern Polytechnical University（电子与信息学院，西北工业大学） ； Shanghai AI Laboratory（上海人工智能实验室） ； Shaanxi Key Laboratory of Information Acquisition and Processing（信息获取与处理陕西省重点实验室） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Journal ref IEEE Transactions on Image Processing 2025

Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation

Jun Wang, Xijuan Zeng, Chunyu Qiang, Ruilong Chen, Shiyao Wang, Le Wang, Wangjing Zhou, Pengfei Cai, Jiahui Zhao, Nan Li, Zihan Li, Yuzhe Liang, Xiaopeng Wang, Haorui Zheng, Ming Wen, Kang Yin, Yiran Wang, Nan Li, Feng Deng, Liang Dong, Chen Zhang, Di Zhang, Kun Gai

机构 * Kuaishou Technology（快手科技） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CL、cs.AI、eess.AS

Cross-attention and Self-attention for Audio-visual Speaker Diarization in MISP-Meeting Challenge

Zhaoyang Li, Haodong Zhou, Longjie Luo, Xiaoxiao Li, Yongxin Chen, Lin Li, Qingyang Hong

机构 * Xiamen University（厦门大学） ； Beijing Jiaotong University（北京交通大学） ； School of Electronic Science and Engineering（电子科学与技术学院） ； School of Electronic Information（电子信息学院） ； School of Informatics（信息学院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)

VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing

Chunyu Qiang, Wang Geng, Yi Zhao, Ruibo Fu, Tao Wang, Cheng Gong, Tianrui Wang, Qiuyu Liu, Jiangyan Yi, Zhengqi Wen, Chen Zhang, Hao Che, Longbiao Wang, Jianwu Dang, Jianhua Tao

机构 * School of New Media and Communication, Tianjin University（新媒体与传播学院，天津大学） ； Kuaishou Technology Co., Ltd.（快手科技有限公司） ； Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University（认知计算与应用重点实验室，智能计算学院，天津大学） ； Department of Automation, Tsinghua University（自动化系，清华大学） ； Beijing National Research Center for Information Science and Technology, Tsinghua University（北京信息科学与技术国家研究中心，清华大学） ； Migu Culture Technology Co., Ltd.（咪咕文化科技有限公司） ； Shenzhen Institute of Advanced Technology, Chinese Academy of Science（深圳先进技术研究院，中国科学院） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Multi-modal brain encoding models for multi-modal stimuli

Subba Reddy Oota, Khushbu Pahwa, Mounika Marreddy, Maneesh Singh, Manish Gupta, Bapi S. Raju

机构 * Technische Universität Berlin（柏林技术大学） ； Rice Univ（莱斯大学） ； Univ of Bonn（波恩大学） ； Spector Inc（Spector公司） ； Microsoft（微软） ； IIIT Hyderabad（海得拉巴国家理工学院） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 26 pages, 15 figures, The Thirteenth International Conference on Learning Representations, ICLR-2025, Singapore. https://openreview.net/pdf?id=0dELcFHig2

Journal ref ICLR-2025, Sinapore

CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment

Edson Araujo, Andrew Rouditchenko, Yuan Gong, Saurabhchand Bhati, Samuel Thomas, Brian Kingsbury, Leonid Karlinsky, Rogerio Feris, James R. Glass, Hilde Kuehne

机构 * Goethe University of Frankfurt（法兰克福歌德大学） ； MIT（麻省理工学院） ； IBM Research（IBM研究院） ； MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室） ； Tuebingen AI Center/University of Tuebingen（图宾根人工智能中心/图宾根大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments To be published at CVPR 2025, code available at https://github.com/edsonroteia/cav-mae-sync

Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustness

Yusheng Zhao, Junyu Luo, Xiao Luo, Weizhi Zhang, Zhiping Xiao, Wei Ju, Philip S. Yu, Ming Zhang

机构 * Peking University（北京大学） ； University of California, Los Angeles（加州大学洛杉矶分校） ； University of Illinois Chicago（伊利诺伊大学香槟分校） ； University of Washington（华盛顿大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Capybara-OMNI: An Efficient Paradigm for Building Omni-Modal Language Models

Xingguang Ji, Jiakang Wang, Hongzhi Zhang, Jingyuan Zhang, Haonan Zhou, Chenxi Sun, Yahui Liu, Qi Wang, Fuzheng Zhang

专题命中 音频语音多模态 ：omni-modal(title)；multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Audio-visual Event Localization on Portrait Mode Short Videos

Wuyang Liu, Yi Chai, Yongpeng Yan, Yanzhen Ren

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Aligned Better, Listen Better for Audio-Visual Large Language Models

Yuxin Guo, Shuailei Ma, Shijie Ma, Xiaoyi Bao, Chen-Wei Xie, Kecheng Zheng, Tingyu Weng, Siyang Sun, Yun Zheng, Wei Zou

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to ICLR 2025

MoMuSE: Momentum Multi-modal Target Speaker Extraction for Real-time Scenarios with Impaired Visual Cues

Junjie Li, Ke Zhang, Shuai Wang, Kong Aik Lee, Man-Wai Mak, Haizhou Li

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMs

Sanjoy Chowdhury, Hanan Gani, Nishit Anand, Sayan Nag, Ruohan Gao, Mohamed Elhoseiny, Salman Khan, Dinesh Manocha

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、eess.AS

Keyword-Oriented Multimodal Modeling for Euphemism Identification

Yuxue Hu, Junsong Li, Meixuan Chen, Dongyu Su, Tongguan Wang, Ying Sha

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Audio-Visual Instance Segmentation

Ruohao Guo, Xianghua Ying, Yaru Chen, Dantong Niu, Guangyao Li, Liao Qu, Yanyu Qi, Jinxing Zhou, Bowei Xing, Wenzhen Yue, Ji Shi, Qixun Wang, Peiliang Zhang, Buwen Liang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by CVPR 2025

NOTA: Multimodal Music Notation Understanding for Visual Large Language Model

Mingni Tang, Jiajia Li, Lu Yang, Zhiqiang Zhang, Jinghao Tian, Zuchao Li, Lefei Zhang, Ping Wang

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、eess.AS

SoundLoc3D: Invisible 3D Sound Source Localization and Classification Using a Multimodal RGB-D Acoustic Camera

Yuhang He, Sangyun Shin, Anoop Cherian, Niki Trigoni, Andrew Markham

专题命中 音频语音多模态 ：multimodal(title)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by WACV2025

Journal ref WACV2025

Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning

Yunbin Tu, Liang Li, Li Su, Qingming Huang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by AAAI 2025

InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions

Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, Qipeng Guo, Haodong Duan, Xin Chen, Han Lv, Zheng Nie, Min Zhang, Bin Wang, Wenwei Zhang, Xinyue Zhang, Jiaye Ge, Wei Li, Jingwen Li, Zhongying Tu, Conghui He, Xingcheng Zhang, Kai Chen, Yu Qiao, Dahua Lin, Jiaqi Wang

专题命中 音频语音多模态 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Github Repo: https://github.com/InternLM/InternLM-XComposer/tree/main/InternLM-XComposer-2.5-OmniLive

YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controls

Zihao Chen, Haomin Zhang, Xinhan Di, Haoyu Wang, Sizhe Shan, Junjie Zheng, Yunming Liang, Yihan Fan, Xinfa Zhu, Wenjie Tian, Yihua Wang, Chaofan Ding, Lei Xie

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 16 pages, 4 figures

Learning Self-Supervised Audio-Visual Representations for Sound Recommendations

Sudha Krishnamurthy

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Published in the Proceedings of the International Symposium on Visual Computing, 2021 https://dl.acm.org/doi/10.1007/978-3-030-90436-4_10

Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignment

Kim Sung-Bin, Arda Senocak, Hyunwoo Ha, Tae-Hyun Oh

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Under-review

Continual Audio-Visual Sound Separation

Weiguo Pian, Yiyang Nan, Shijian Deng, Shentong Mo, Yunhui Guo, Yapeng Tian

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments NeurIPS 2024

Getting More for Less: Using Weak Labels and AV-Mixup for Robust Audio-Visual Speaker Verification

Anith Selvakumar, Homa Fashandi

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to INTERSPEECH 2024

Journal ref Proc. Interspeech 2024, 4728-4732