多模态大模型 - arXivDaily 专题

2312.02781 2023-12-06 cs.CV cs.AI 88%

PMMTalk: Speech-Driven 3D Facial Animation from Complementary Pseudo Multi-modal Features

Tianshun Han, Shengnan Gui, Yiqing Huang, Baihui Li, Lijian Liu, Benjia Zhou, Ning Jiang, Quan Lu, Ruicong Zhi, Yanyan Liang, Du Zhang, Jun Wan

专题命中音频语音多模态：multi-modal(title,abstract)；cross-modal(abstract)；image-text(abstract)；audio-visual(abstract)

2602.03570 2026-07-28 cs.LG 版本更新 88%

Asymmetric Hierarchical Anchoring for Robust Audio-Visual Cross-Modal Generalization

非对称层次锚定用于音频-视觉联合表示：解决信息分配模糊性以实现鲁棒跨模态泛化

Bixing Wu, Yuhong Zhao, Zongli Ye, Jiachen Lian, Xiangyu Yue, Gopala Anumanchipalli

机构 * Zhejiang University, China（浙江大学）； University of California, Berkeley, USA（加州大学伯克利分校）； MMLab, Chinese University of Hong Kong, China（香港中文大学MMLab）

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title,abstract)

AI总结本文提出非对称层次锚定框架，通过结构化语义锚点解决跨模态泛化中的信息分配模糊性，提升语义一致性和表示纯度。

2501.18314 2025-07-15 cs.MM cs.CV cs.SD eess.AS 88%

AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment

Yuqin Cao, Xiongkuo Min, Yixuan Gao, Wei Sun, Guangtao Zhai

机构 * Institute of Image Communication ； Network Engineering, Shanghai Key Laboratory of Digital Media Processing ； Transmissions, Shanghai Jiao Tong University, Shanghai ； School of Communication \& Electronic Engineering, East China Normal University, Shanghai

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(title)；分类 cs.CV、cs.MM、eess.AS

2411.01991 2024-11-05 eess.SP 88%

Multimodal Trustworthy Semantic Communication for Audio-Visual Event Localization

Yuandi Li, Zhe Xiang, Fei Yu, Zhangshuang Guan, Hui Ji, Zhiguo Wan, Cheng Feng

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)

2207.04213 2023-03-07 cs.MM cs.CV cs.LG cs.SD eess.AS 88%

Dual-Path Cross-Modal Attention for better Audio-Visual Speech Extraction

Zhongweiyang Xu, Xulin Fan, Mark Hasegawa-Johnson

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(title)；分类 cs.CV、cs.MM、eess.AS

Comments Paper Accepted by ICASSP2023

1909.13714 2019-10-01 cs.MM cs.CL cs.CV 88%

Towards Multimodal Understanding of Passenger-Vehicle Interactions in Autonomous Vehicles: Intent/Slot Recognition Utilizing Audio-Visual Data

Eda Okur, Shachi H Kumar, Saurav Sahay, Lama Nachman

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title)；分类 cs.CV、cs.CL、cs.MM

Comments Presented as a short-paper at the 23rd Workshop on the Semantics and Pragmatics of Dialogue (SemDial 2019 - LondonLogue), Sep 4-6, 2019, London, UK

Journal ref Proceedings of the 23rd Workshop on the Semantics and Pragmatics of Dialogue (SEMDIAL), pp. 213-215, London, United Kingdom, September 2019

1412.3121 2016-02-19 cs.NE cs.LG 88%

Multimodal Transfer Deep Learning with Applications in Audio-Visual Recognition

Seungwhan Moon, Suyoun Kim, Haohan Wang

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title,abstract)

Comments 6 pages, MMML workshop at NIPS 2015

2602.07064 2026-04-08 cs.CV 87%

OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimization

OmniFysics：通过多模态信号处理和网络优化实现物理智能进化

Minghao Han, Dingkang Yang, Yue Jiang, Yizhou Liu, Lihua Zhang

机构 * College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）； Fysics AI

专题命中音频语音多模态：omni-modal(title,abstract)；multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)

AI总结 OmniFysics通过统一图像、音频、视频和文本的多模态信号处理，结合动态物理数据引擎和物理知识注入，实现网络化AI系统的自主优化，提升物理理解能力。

Comments This work has been submitted to the IEEE for possible publication

2602.06139 2026-02-09 cs.CV 87%

EgoAVU: Egocentric Audio-Visual Understanding

EgoAVU：第一人称音频视觉理解

Ashish Seth, Xinhao Mei, Changsheng Zhao, Varun Nagaraja, Ernie Chang, Gregory P. Meyer, Gael Le Lan, Yunyang Xiong, Vikas Chandra, Yangyang Shi, Dinesh Manocha, Zhipeng Cai

机构 * Meta ； University of Maryland, College Park（马里兰大学College Park分校）

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；multi-modal(abstract)；cross-modal(abstract)

AI总结 EgoAVU通过生成第一人称音频视觉叙述和问题，提升多模态大语言模型在第一人称视频理解中的性能。

2602.03892 2026-02-05 cs.CV cs.AI cs.LG cs.MM cs.SD eess.AS 87%

Audit After Segmentation: Reference-Free Mask Quality Assessment for Language-Referred Audio-Visual Segmentation

分段后审计：用于语言指代音频视频分段的无参考掩码质量评估

Jinxing Zhou, Yanghao Zhou, Yaoting Wang, Zongyan Han, Jiaqi Ma, Henghui Ding, Rao Muhammad Anwer, Hisham Cholakkal

机构 * MBZUAI ； National University of Singapore（国立新加坡大学）； Fudan University（复旦大学）

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI、cs.MM

AI总结本研究提出 MQA-RefAVS 任务，通过多模态大语言模型评估语言指代音频视频分段中掩码质量，提升分割准确性与可解释性。

2506.11436 2025-12-11 cs.CV 87%

TAViS: Text-bridged Audio-Visual Segmentation with Foundation Models

TAViS: 基于文本的音频视觉分割与基础模型

Ziyang Luo, Nian Liu, Xuguang Yang, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan, Junwei Han

机构 * Northwestern Polytechnical University（西北工业大学）； Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）； Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合国家科学中心人工智能研究院）

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；multimodal foundation model(abstract)

AI总结 TAViS通过文本桥接机制结合多模态基础模型与分割模型，实现高效的音频视觉分割与跨模态对齐。

Comments ICCV2025,code:https://github.com/Sssssuperior/TAViS

2511.22826 2025-12-04 cs.CV 87%

Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs

某些模态比其他模态更平等：在MLLMs中解码和架构多模态整合

Tianle Chen, Chaitanya Chakka, Arjun Reddy Akula, Xavier Thomas, Deepti Ghadiyaram

机构 * Boston University（波士顿大学）； Google DeepMind（谷歌DeepMind）

专题命中音频语音多模态：multimodal(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；audio-visual(abstract)

AI总结本文研究了多模态大语言模型对矛盾模态的鲁棒性，提出模态对齐调优策略以提升多模态推理能力。

2509.17537 2025-09-24 cs.CV 87%

SimToken: A Simple Baseline for Referring Audio-Visual Segmentation

Dian Jin, Yanghao Zhou, Jinxing Zhou, Jiaqi Ma, Ruohao Guo, Dan Guo

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；MLLM(abstract)；cross-modal(abstract)

Comments Project page: https://github.com/DianJin-HFUT/SimToken

2505.08294 2025-06-17 cs.CV 87%

FauForensics: Boosting Audio-Visual Deepfake Detection with Facial Action Units

Jian Wang, Baoyuan Wu, Li Liu, Qingshan Liu

机构 * School of Computer Science, Nanjing University of Posts and Telecommunications（南京邮电大学计算机科学学院）； School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）； The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；multi-modal(abstract)；cross-modal(abstract)

2505.06803 2025-05-13 cs.SD cs.CL cs.CV cs.MM eess.AS 87%

Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation

Xilin Jiang, Junkai Wu, Vishal Choudhari, Nima Mesgarani

专题命中音频语音多模态：cross-modal(title,abstract)；multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL、cs.MM

2406.10591 2024-06-18 eess.AS cs.AI cs.CV cs.MM cs.SD 87%

MINT: a Multi-modal Image and Narrative Text Dubbing Dataset for Foley Audio Content Planning and Generation

Ruibo Fu, Shuchen Shi, Hongming Guo, Tao Wang, Chunyu Qiang, Zhengqi Wen, Jianhua Tao, Xin Qi, Yi Lu, Xiaopeng Wang, Zhiyong Wang, Yukun Liu, Xuefei Liu, Shuai Zhang, Guanjun Li

专题命中音频语音多模态：multi-modal(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

2405.07451 2024-05-14 cs.CV 87%

CLIP-Powered TASS: Target-Aware Single-Stream Network for Audio-Visual Question Answering

Yuanyuan Jiang, Jianqin Yin

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；image-text(abstract)

Comments Submitted to the Journal on February 6, 2024

2608.07923 2026-08-11 cs.CV cs.MM cs.SD 新提交 87%

SCoPE: Training-Free Audio-Visual Event Perception via Sparse Cross-Modal Prior Exchange

SCoPE：基于稀疏跨模态先验交换的无训练视听事件感知

Jaemo Jeong, Junho Yoon, Hyunju Kim, Dongman Lee

机构 * KAIST（韩国科学技术院）

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(title)；分类 cs.CV、cs.MM

AI总结针对无训练视听事件感知的虚假共激活问题，提出SCoPE框架，通过跨模态标签竞争消除FCA，在LLP等数据集上显著提升性能且可直接迁移。

Comments 17 pages, 6 figures

2504.00837 2026-03-09 cs.SD cs.AI cs.MM 87%

A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives

从单模态、跨模态和多模态视角综述音乐生成

Shuyu Li, Shulei Ji, Zihao Wang, Songruoyao Wu, Jiaxing Yu, Kejun Zhang

机构 * College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）； Innovation Center of Yangtze River Delta, Zhejiang University（浙江大学长三角创新中心）

专题命中音频语音多模态：multi-modal(title,abstract)；cross-modal(title)；分类 cs.AI、cs.MM

AI总结本文综述了多模态音乐生成的研究现状，探讨了多模态数据对齐、系统评估方法及未来研究方向。

2510.21797 2026-02-11 cs.LG cs.AI cs.SD eess.AS 87%

Quantifying Multimodal Imbalance: A GMM-Guided Adaptive Loss for Audio-Visual Learning

量化多模态不平衡：一种基于GMM的自适应损失用于音频-视觉学习

Zhaocheng Liu, Zhiwen Yu, Xiaoqing Liu

机构 * South China University of Technology（南方科技大学）

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title)；分类 cs.AI、eess.AS

AI总结本文提出基于GMM的自适应损失，用于量化和缓解多模态学习中的样本级不平衡问题，通过双阶段框架提升音频-视觉学习性能。

2506.04635 2025-06-06 cs.CL cs.CV 87%

ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition

Thai-Binh Nguyen, Thi Van Nguyen, Quoc Truong Do, Chi Mai Luong

机构 * New Turing Institute（新图灵研究所）； Phenikaa University（费尼卡大学）； Vietnam Institute of Information Technology（越南信息技术研究所）； VietAI Research（越南人工智能研究）

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(title)；分类 cs.CV、cs.CL

Comments Accepted at Interspeech 2025

2405.00574 2025-02-05 cs.CV cs.MM 87%

EALD-MLLM: Emotion Analysis in Long-sequential and De-identity videos with Multi-modal Large Language Model

Deng Li, Xin Liu, Bohao Xing, Baiqiang Xia, Yuan Zong, Bihan Wen, Heikki Kälviäinen

专题命中音频语音多模态：multi-modal(title)；MLLM(title)；multimodal(abstract)；分类 cs.CV、cs.MM

2404.11938 2024-04-19 cs.MM cs.DC cs.SD eess.AS 87%

HyDiscGAN: A Hybrid Distributed cGAN for Audio-Visual Privacy Preservation in Multimodal Sentiment Analysis

Zhuojia Wu, Qi Zhang, Duoqian Miao, Kun Yi, Wei Fan, Liang Hu

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title)；分类 cs.MM、eess.AS

Comments 13 pages, IJCAI-2024

2307.16620 2023-08-02 cs.SD cs.CV eess.AS 87%

Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics

Chen Liu, Peike Li, Xingqun Qi, Hu Zhang, Lincheng Li, Dadong Wang, Xin Yu

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(title)；分类 cs.CV、eess.AS

Comments This paper has been received by ACM MM 23

2301.10180 2023-01-25 cs.CL cs.SD eess.AS 87%

A Multi-Purpose Audio-Visual Corpus for Multi-Modal Persian Speech Recognition: the Arman-AV Dataset

Javad Peymanfard, Samin Heydarian, Ali Lashini, Hossein Zeinali, Mohammad Reza Mohammadi, Nasser Mozayani

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(title)；分类 cs.CL、eess.AS

2203.05178 2022-03-11 cs.CV cs.SD eess.AS 87%

An Audio-Visual Attention Based Multimodal Network for Fake Talking Face Videos Detection

Ganglai Wang, Peng Zhang, Lei Xie, Wei Huang, Yufei Zha, Yanning Zhang

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(title)；分类 cs.CV、eess.AS

2110.12136 2022-03-07 cs.CV cs.SD eess.AS eess.IV 87%

A Study of Multimodal Person Verification Using Audio-Visual-Thermal Data

Madina Abdrakhmanova, Saniya Abushakimova, Yerbolat Khassanov, Huseyin Atakan Varol

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title)；分类 cs.CV、eess.AS

Comments 7 pages, 4 figures, 4 tables

2111.05890 2021-11-12 cs.CV cs.SD eess.AS 87%

Multimodal End-to-End Group Emotion Recognition using Cross-Modal Attention

Lev Evtodienko

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(title)；分类 cs.CV、eess.AS

1809.08001 2020-11-05 cs.CV cs.SD eess.AS 87%

Perfect match: Improved cross-modal embeddings for audio-visual synchronisation

Soo-Whan Chung, Joon Son Chung, Hong-Goo Kang

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(title)；分类 cs.CV、eess.AS

Comments Preprint. Work in progress

1909.07352 2020-07-15 eess.AS cs.SD eess.SP 87%

Audio-Visual Speech Separation and Dereverberation with a Two-Stage Multimodal Network

Ke Tan, Yong Xu, Shi-Xiong Zhang, Meng Yu, Dong Yu

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title)；分类 eess.AS；multi-modal(comments)

Comments 13 pages, accepted by IEEE JSTSP Special Issue on Deep Learning for Multi-modal Intelligence across Speech, Language, Vision, and Heterogeneous Signals

1. 音频语音多模态 4567 篇