多模态大模型

Absolute Geometry Calibration of Distributed Microphone Arrays in an Audio-Visual Sensor Network

Florian Jacob, Reinhold Haeb-Umbach

专题命中 音频语音多模态 ：audio-visual(title,abstract)

Automatic Summarization of Soccer Highlights Using Audio-visual Descriptors

Arnau Raventos, Raul Quijada, Luis Torres, Francesc Tarres

专题命中 音频语音多模态 ：audio-visual(title,abstract)

A Topic Model Approach to Multi-Modal Similarity

Rasmus Troelsgård, Bjørn Sand Jensen, Lars Kai Hansen

专题命中 音频语音多模态 ：multi-modal(title,abstract)

Comments topic modelling workshop at NIPS 2013

Tableaux for multi-modal hybrid logic with binders, transitive relations and relation hierarchies

M. Cialdea Mayer

专题命中 音频语音多模态 ：multi-modal(title,abstract)

Comments This paper has been withdrawn by the author because it is superseded by a new one: A Proof Procedure for Hybrid Logic with Binders, Transitivity and Relation Hierarchies(extended version)

Information Optimization in Coupled Audio-Visual Cortical Maps

Mehran Kardar, A. Zee

专题命中 音频语音多模态 ：audio-visual(title,abstract)

Comments 7 pages, 1 figure

Supporting Out-of-turn Interactions in a Multimodal Web Interface

Atul Shenoy, Naren Ramakrishnan, Manuel A. Perez-Quinones, Srinidhi Varadarajan

专题命中 音频语音多模态 ：multimodal(title,abstract)

Parameter- and Bandwidth-Efficient Edge--cloud Many-to-Many Speech-to-Text Translation

带宽高效且隐私保护的边缘-云多对多语音翻译

Yexing Du, Kaiyuan Liu, Youcheng Pan, Bo Yang, Lei Chen, Ming Liu, Bing Qin, Yang Xiang

机构 * Harbin Institute of Technology（哈尔滨工业大学） ； Pengcheng Laboratory（鹏城实验室） 纠错

专题命中 音频语音多模态 ：MLLM(abstract,abstract_cn)；multimodal(abstract)；分类 cs.AI

AI总结 提出边缘-云协同框架ESRT，通过分割推理架构压缩中间特征实现带宽降低10倍和语音隐私保护，并采用多任务加权课程学习策略实现45种语言的多对多语音翻译。

Identity as Presence: Towards Appearance and Voice Personalized Joint Audio-Video Generation

身份作为存在：迈向基于外观和声音的联合音频视频生成

Qin Chen, Yingjie Chen, Shilun Lin, Cai Xing, Binxin Yang, Long Zhou, Qixin Yan, Wenjing Wang, Dingming Liu, Hao Liu, Chen Li, Jing Lyu

机构 * Tencent Inc.（腾讯公司） ； Peking University（北京大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

AI总结 本文提出一个统一的框架，实现身份感知的音频视频生成，通过数据整理管道和灵活的身份注入机制，提升生成内容的高保真度和一致性。

OmniVR: Joint Video-Audio Conditional Generation for Restoring Degraded Historical Films

OmniVR：用于恢复退化历史影片的联合视频-音频条件生成模型

Xin Lu, Zihao Fan, Mingchen Zhong, Jie Huang, Xueyang Fu, Zheng-Jun Zha

机构 * University of Science and Technology of China（中国科学技术大学） ； JD Explore Academy（京东探索研究院） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

AI总结 提出首个联合音视频生成修复模型OmniVR，通过三项关键设计解决历史影片的音视频共同退化问题，在多类指标上超越现有方法，还发布了相关基准OmniVRBench

Bridging the Age Gap: Towards Detecting Neural Audio Codec Synthesized Elderly Speech Deepfake

弥合年龄差距：面向检测神经音频编解码器合成的老年语音深度伪造

Orchid Chetia Phukan, Girish, Mohd Mujtaba Akhtar, Chi-Chun Lee

机构 * BIIC Lab, NTHU, Taiwan（国立台湾大学BIIC实验室） ； UPES, India（印度UPES大学） ； VBSPU, India（印度VBSPU大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；multimodal foundation model(abstract)；分类 eess.AS

AI总结 提出老年语音编解码伪造检测（ECFD）任务，发布中英文ECF数据集，发现现有检测器泛化差，利用多模态基础模型融合（BONSAI框架）实现1.66%平均等错误率，建立新基准。

Comments Accepted to INTERSPEECH 2026

Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation

Baton: 用于联合视频-音频生成的显式语义蓝图

Shuyuan Tu, Qi Tian, Zihan Yang, Yue Wu, Xintong Han, Weijie Kong, Jiangfeng Xiong, Jian-Wei Zhang, Zhao Zhong, Liefeng Bo, Zuxuan Wu, Yu-Gang Jiang

机构 * Fudan University（复旦大学） ； Tencent Hunyuan（腾讯幻元） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

AI总结 提出Baton框架，通过VA-Planner生成语义对齐的模态感知规划令牌作为蓝图，注入扩散骨干以协调视频和音频去噪，解决现有方法因缺乏共享长期规划导致的跨模态对齐脆弱问题。

Escape the Language Prior: Mitigating Late-Stage Modality Collapse in Audio Reasoning via Modality-Aware Policy Optimization

逃离语言先验：通过模态感知策略优化缓解音频推理中的后期模态崩溃

Cihan Xiao, Yiwen Shao, Chenxing Li, Xiang He, Zhenwen Liang, Steve Yves, Sanjeev Khudanpur, Liefeng Bo

机构 * Johns Hopkins University（约翰霍普金斯大学） ； Tencent Hunyuan（腾讯文言） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；omni-modal(abstract)；分类 cs.CL

AI总结 针对多模态大语言模型在强化学习后训练中因统一策略梯度忽略模态依赖性而导致的后期模态崩溃问题，提出模态感知策略优化（MAPO）框架，通过模态相关性掩码和辅助注意力损失分支动态聚焦梯度并维持跨模态推理，在复杂音频推理基准上取得新最优结果。

OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models

OmniZip：面向快速多模态大语言模型的音频引导动态令牌压缩

Keda Tao, Kele Shao, Bohan Yu, Weiqiang Wang, Jian liu, Huan Wang

机构 * Zhejiang University（浙江大学） ； Westlake University（西湖大学） ； Ant Group（蚂蚁集团） ； Shanghai Innovation Institute（上海创新研究院） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

AI总结 本文提出OmniZip，一种无需训练的音频引导多模态令牌压缩框架，通过动态压缩音频视频令牌提升推理速度和内存效率，保持模型性能。

Comments [CVPR 2026] Code Link: https://github.com/KD-TAO/OmniZip

CineSRD: Leveraging Visual, Acoustic, and Linguistic Cues for Open-World Visual Media Speaker Diarization

CineSRD：利用视觉、听觉和语言线索进行开放世界视觉媒体说话人聚类

Liangbin Huang, Xiaohua Liao, Chaoqun Cui, Shijing Wang, Zhaolong Huang, Yanlong Du, Wenji Mao

机构 * Hujing Digital Media and Entertainment Group（华景数字媒体与娱乐集团） ； MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS部门） ； School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院） ； School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

AI总结 CineSRD通过整合视觉、听觉和语言线索，解决开放世界视觉媒体中说话人识别的挑战，提出统一多模态框架并构建专用基准数据集，验证其在复杂场景下的鲁棒性和泛化能力。

Comments Accepted to CVPR 2026

MOVA: Towards Scalable and Synchronized Video-Audio Generation

MOVA：迈向可扩展且同步的视频-音频生成

OpenMOSS Team, Donghua Yu, Mingshu Chen, Qi Chen, Qi Luo, Qianyi Wu, Qinyuan Cheng, Ruixiao Li, Tianyi Liang, Wenbo Zhang, Wenming Tu, Xiangyu Peng, Yang Gao, Yanru Huo, Ying Zhu, Yinze Luo, Yiyang Zhang, Yuerong Song, Zhe Xu, Zhiyu Zhang, Chenchen Yang, Cheng Chang, Chushu Zhou, Hanfu Chen, Hongnan Ma, Jiaxi Li, Jingqi Tong, Junxi Liu, Ke Chen, Shimin Li, Shiqi Jiang, Songlin Wang, Wei Jiang, Zhaoye Fei, Zhiyuan Ning, Chunguo Li, Chenhui Li, Ziwei He, Zengfeng Huang, Xie Chen, Xipeng Qiu

专题命中 音频语音多模态 ：multimodal(abstract)；image-text(abstract)；audio-visual(abstract)；分类 cs.CV

AI总结 MOVA通过混合专家架构生成高质量同步音频视频内容，支持图像-文本到视频-音频的生成任务，推动开放研究与创作者社区发展。

Comments Technical report for MOVA (open-source video-audio generation model). 38 pages, 10 figures, 22 tables. Project page: https://mosi.cn/models/mova Code: https://github.com/OpenMOSS/MOVA Models: https://huggingface.co/collections/OpenMOSS-Team/mova. Qinyuan Cheng and Tianyi Liang are project leader. Xie Chen and Xipeng Qiu are corresponding authors

Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization

Yanhao Jia, Ji Xie, S Jivaganesh, Hao Li, Xu Wu, Mengmi Zhang

机构 * Nanyang Technological University, Singapore（南洋理工大学） ； Peking University, China（北京大学） ； Shenzhen University, China（深圳大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments NeurIPS 2025, Spotlight

Qwen3-Omni Technical Report

Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo Zheng, Rui Men, Fan Zhou, Bowen Yu, Jianxin Yang, Le Yu, Jingren Zhou, Junyang Lin

机构 * Qwen Team（通义实验室） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments https://github.com/QwenLM/Qwen3-Omni

Spatial and Semantic Embedding Integration for Stereo Sound Event Localization and Detection in Regular Videos

Davide Berghi, Philip J. B. Jackson

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 eess.AS

Bimodal Connection Attention Fusion for Speech Emotion Recognition

Jiachen Luo, Huy Phan, Lin Wang, Joshua D. Reiss

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI、cs.MM

DocVideoQA: Towards Comprehensive Understanding of Document-Centric Videos through Question Answering

Haochen Wang, Kai Hu, Liangcai Gao

专题命中 音频语音多模态 ：multimodal(abstract)；MLLM(abstract)；audio-visual(abstract)；分类 cs.CV

A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models

Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, Aman Chadha

专题命中 音频语音多模态 ：multimodal(abstract)；multimodal foundation model(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments EMNLP 2024 Findings

It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition

Chen Chen, Ruizhe Li, Yuchen Hu, Sabato Marco Siniscalchi, Pin-Yu Chen, Ensiong Chng, Chao-Han Huck Yang

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Accepted to ICLR 2024, 17 pages. This work will be open sourced under MIT license

GPT-4V(ision) as A Social Media Analysis Engine

Hanjia Lyu, Jinfa Huang, Daoan Zhang, Yongsheng Yu, Xinyi Mou, Jinsheng Pan, Zhengyuan Yang, Zhongyu Wei, Jiebo Luo

专题命中 音频语音多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality

Wei-Ning Hsu, Bowen Shi

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2022

mSLAM: Massively multilingual joint pre-training for speech and text

Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, Alexis Conneau

专题命中 音频语音多模态 ：multimodal(abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CL

Spoken ObjectNet: A Bias-Controlled Spoken Caption Dataset

Ian Palmer, Andrew Rouditchenko, Andrei Barbu, Boris Katz, James Glass

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Presented at Interspeech 2021. This version contains additional experiments on the Spoken ObjectNet test set

Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision

Abhinav Shukla, Stavros Petridis, Maja Pantic

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted at the Workshop on Self-supervision in Audio and Speech at ICML 2020

LEGATO 2: Toward Multimodal Sheet Music Recognition and Understanding

LEGATO 2：迈向多模态乐谱识别与理解

Guang Yang, Brian Siyuan Zheng, Victoria Ebert, Noah A. Smith

机构 * Paul G. Allen School of Computer Science & Engineering, University of Washington（保罗·G·艾伦计算机科学与工程学院，华盛顿大学） 纠错

专题命中 音频语音多模态 ：multimodal(title)；分类 cs.CV、cs.AI

AI总结 LEGATO 2提出从乐谱图像提取符号与语义知识的新流程，结合系统级分割和自回归视觉语言模型，能生成含文本的符号转录，在多数据集上优于现有技术，还能补充语言模型视觉输入，提升乐谱理解水平。

Comments 23 pages. Equal contribution: Guang Yang and Brian Siyuan Zheng

Are you speaking my languages? On spoken language adherence in multimodal LLMs

你在说我的语言吗？多模态大语言模型中的口语遵循问题

Hyungwon Kim, Kandarp Joshi, Lillian Zhou, Pavel Golik, Petar Aleksic

机构 * Google DeepMind（谷歌DeepMind） 纠错

专题命中 音频语音多模态 ：multimodal(title)；分类 cs.CL、eess.AS

AI总结 针对多模态大语言模型在自动语音识别中输出语言识别错误的问题，提出软提示方法、监督微调和思维链推理三种缓解策略，并引入新指标量化语言违背，比较各方法在减少违规和保持ASR性能上的效果。

Comments 7 pages, 3 tables in the main body

Few-shot Acoustic Synthesis with Multimodal Flow Matching

少样本声学合成与多模态流匹配

Amandine Brunetto

机构 * Center for Robotics, Mines Paris - PSL University（机器人中心，巴黎 Mines - PSL 大学） 纠错

专题命中 音频语音多模态 ：multimodal(title)；分类 cs.CV、eess.AS

AI总结 本文提出FLAC方法，通过流匹配生成少样本声学合成，结合空间、几何和声学线索生成新的场景房间脉冲响应，优于现有八样本基线。

Comments To appear at CVPR 2026. 23 pages, 16 figures. Project Page: https://amandinebtto.github.io/FLAC/