多模态大模型

Exploring Human-AI Complementarity in CPS Diagnosis Using Unimodal and Multimodal BERT Models

Kester Wong, Sahan Bulathwela, Mutlu Cukurova

机构 * UCL Knowledge Lab, Institute of Education, University College London, UK（伦敦大学学院教育研究所知识实验室） ； UCL Centre for Artificial Intelligence, Department of Computer Science, University College London, UK（伦敦大学学院人工智能中心计算机科学系） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted to appear in the workshop proceedings for the HEXED'25 workshop in the 26th International Conference on Artificial Intelligence in Education 2025 (AIED 2025), 22 July 2025, Palermo, Italy. 5 pages

CorMulT: A Semi-supervised Modality Correlation-aware Multimodal Transformer for Sentiment Analysis

Yangmin Li, Ruiqi Zhu, Wengen Li

机构 * Information Networking Institute at Carnegie Mellon University (CMU)（卡内基梅隆大学信息网络研究所） ； School of Computer Science and Technology at Tongji University（同济大学计算机科学与技术学院） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning

Siddharth Srivastava, Gaurav Sharma

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Journal ref CVPR 2024

Cross-Modal Watermarking for Authentic Audio Recovery and Tamper Localization in Synthesized Audiovisual Forgeries

Minyoung Kim, Sehwan Park, Sungmin Cha, Paul Hongsuck Seo

机构 * Korea University（韩国大学） ； New York University（纽约大学） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；分类 cs.MM、eess.AS

Comments 5 pages, 2 figures, Interspeech 2025

Language-Guided Contrastive Audio-Visual Masked Autoencoder with Automatically Generated Audio-Visual-Text Triplets from Videos

Yuchi Ishikawa, Shota Nakada, Hokuto Munakata, Kazuhiro Saito, Tatsuya Komatsu, Yoshimitsu Aoki

机构 * LY Corporation（LY公司） ； Keio University（庆应大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Interspeech 2025

SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generation

Youliang Zhang, Zhaoyang Li, Duomin Wang, Jiahe Zhang, Deyu Zhou, Zixin Yin, Xili Dai, Gang Yu, Xiu Li

机构 * Tsinghua University（清华大学） ； StepFun ； The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）） ； The Hong Kong University of Science and Technology（香港科技大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Audio-Visual Speech Separation via Bottleneck Iterative Network

Sidong Zhang, Shiv Shankar, Trang Nguyen, Andrea Fanelli, Madalina Fiterau

机构 * Manning College of Information \& Computer Sciences, University of Massachusetts Amherst, Amherst, U.S. ； Dolby Laboratories, San Francisco, U.S. 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.MM、eess.AS

Comments Accepted to the 42nd International Conference on Machine Learning Workshop on Machine Learning for Audio

Towards an Automated Multimodal Approach for Video Summarization: Building a Bridge Between Text, Audio and Facial Cue-Based Summarization

Md Moinul Islam, Sofoklis Kakouros, Janne Heikkilä, Mourad Oussalah

机构 * Center for Machine Vision and Signal Analysis, Faculty of ITEE, University of Oulu, Finland（机器视觉与信号分析中心，ITEE学院，奥卢大学，芬兰） ； Center for Machine Vision（机器视觉与信号分析中心） ； Signal Analysis, Faculty of ITEE, University of Oulu, Finland（信号分析，ITEE学院，奥卢大学，芬兰） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted to HHAI WS 2025: Workshops at the Fourth International Conference on Hybrid Human-Artificial Intelligence (HHAI)

Efficiently Serving Large Multimodal Models Using EPD Disaggregation

Gursimran Singh, Xinglu Wang, Yifan Hu, Timothy Yu, Linzi Xing, Wei Jiang, Zhefeng Wang, Xiaolong Bai, Yi Li, Ying Xiong, Yong Zhang, Zhenan Fan

机构 * Huawei Technologies Canada, BC, Canada（华为加拿大技术有限公司） ； Simon Fraser University, BC, Canada（西蒙弗雷泽大学） ； Huawei Cloud, China（华为云） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments 17 pages, 12 figures, 9 tables

Journal ref International Conference on Machine Proceedings of the 42nd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025

SSAVSV: Towards Unified Model for Self-Supervised Audio-Visual Speaker Verification

Gnana Praveen Rajasekhar, Jahangir Alam

机构 * Computer Research Institute of Montreal (CRIM), Canada（蒙特利尔计算机研究院（CRIM）） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction

Wenxuan Wu, Shuai Wang, Xixin Wu, Helen Meng, Haizhou Li

机构 * Department of SEEM（SEEM系） ； SRIBD, School of Data Science（数据科学学院） ； School of Intelligence Science and Technology（智能科学与技术学院） ； Department of ECE（电子工程系） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.MM、eess.AS

Comments Accepted by Interspeech 2025

Multimodal Emotion Coupling via Speech-to-Facial and Bodily Gestures in Dyadic Interaction

Von Ralph Dane Marquez Herbuela, Yukie Nagai

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.MM、eess.AS

CoMuMDR: Code-mixed Multi-modal Multi-domain corpus for Discourse paRsing in conversations

Divyaksh Shukla, Ritesh Baviskar, Dwijesh Gohil, Aniket Tiwari, Atul Shree, Ashutosh Modi

机构 * Indian Institute of Technology Kanpur (IIT Kanpur)（印度理工学院坎浦尔学院） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted at ACL Findings 2025 (16 pages: 5 pages main content + 3 pages references + 8 pages appendix)

GAME: Learning Multimodal Interactions via Graph Structures for Personality Trait Estimation

Kangsheng Wang, Yuhang Li, Chengwei Ye, Yufei Lin, Huanzhen Zhang, Bohan Hu, Linuo Xu, Shuyan Liu

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments The article contains serious scientific errors and cannot be corrected by updating the preprint

MELT: Towards Automated Multimodal Emotion Data Annotation by Leveraging LLM Embedded Knowledge

Xin Jing, Jiadong Wang, Iosif Tsangko, Andreas Triantafyllopoulos, Björn W. Schuller

机构 * Technical University of Munich（慕尼黑技术大学） ； Imperial College London（伦敦帝国理工学院） ； CHI – Chair of Health Informatics（健康信息学系） ； Munich Centre for Machine Learning（慕尼黑机器学习中心） ； GLAM – Group on Language, Audio, & Music（语言、音频与音乐小组） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI、eess.AS

Plug-and-Play Co-Occurring Face Attention for Robust Audio-Visual Speaker Extraction

Zexu Pan, Shengkui Zhao, Tingting Wang, Kun Zhou, Yukun Ma, Chong Zhang, Bin Ma

机构 * Alibaba Group（阿里巴巴集团） ； Nanjing University of Posts and Telecommunications（南京邮电大学） ； Tongyi Lab（通义实验室） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.AI、eess.AS

Comments Interspeech 2025

MPE-TTS: Customized Emotion Zero-Shot Text-To-Speech Using Multi-Modal Prompt

Zhichao Wu, Yueteng Kang, Songjun Cao, Long Ma, Qiulin Li, Qun Yang

机构 * Nanjing University of Aeronautics and Astronautics（南京航空航天大学） ； TencentChina（腾讯中国） ； Youtu Lab（你图实验室） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.AI、eess.AS

Comments Accepted by InterSpeech

Adversarial Deep Metric Learning for Cross-Modal Audio-Text Alignment in Open-Vocabulary Keyword Spotting

Youngmoon Jung, Yong-Hyeok Lee, Myunghun Jung, Jaeyoung Roh, Chang Woo Han, Hoon-Young Cho

机构 * Samsung Research（三星研究院） 纠错

专题命中 音频语音多模态 ：cross-modal(title)；multi-modal(abstract)；分类 cs.AI、eess.AS

Comments 5 pages, 1 figure, Accepted at Interspeech 2025

MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition

Sungnyun Kim, Kangwook Jang, Sangmin Bae, Sungwoo Cho, Se-Young Yun

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted to ICML 2025

A Survey on Multimodal Music Emotion Recognition

Rashini Liyanarachchi, Aditya Joshi, Erik Meijering

机构 * University of New South Wales（新南威尔士大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.MM、eess.AS

Steganography Beyond Space-Time with Chain of Multimodal AI

Ching-Chun Chang, Isao Echizen

机构 * Information and Society Research Division, National Institute of Informatics（信息与社会研究部，日本信息机构） ； Graduate School of Information Science and Technology, University of Tokyo（东京大学信息科学与技术研究生院） ； School of Multidisciplinary Sciences, Graduate University for Advanced Studies (SOKENDAI)（多学科科学学院，研究生高级研究大学（SOKENDAI）） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI、cs.MM

Journal ref Scientific Reports, vol. 15, no. 1, Article 12908, 2025

MusFlow: Multimodal Music Generation via Conditional Flow Matching

Jiahao Song, Yuzhao Wang

机构 * School of Mathematics and Statistics, Shanxi University（山西大学数学与统计学院） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.MM、eess.AS

Audio-Visual Speaker Tracking: Progress, Challenges, and Future Directions

Jinzheng Zhao, Yong Xu, Xinyuan Qian, Davide Berghi, Peipei Wu, Meng Cui, Jianyuan Sun, Philip J. B. Jackson, Wenwu Wang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.MM、eess.AS

GazeLLM: Multimodal LLMs incorporating Human Visual Attention

Jun Rekimoto

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Journal ref Augmented Humans 2025

Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attention

Yuzhe Weng, Haotian Wang, Tian Gao, Kewei Li, Shutong Niu, Jun Du

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

United we stand, Divided we fall: Handling Weak Complementary Relationships for Audio-Visual Emotion Recognition in Valence-Arousal Space

R. Gnana Praveen, Jahangir Alam, Eric Charton

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Achieved 2nd place in valence arousal challenge Submission to CVPR2025 Workshop

UniSync: A Unified Framework for Audio-Visual Synchronization

Tao Feng, Yifan Xie, Xun Guan, Jiyuan Song, Zhou Liu, Fei Ma, Fei Yu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments 7 pages, 3 figures, accepted by ICME 2025

Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Control

Hejia Chen, Haoxian Zhang, Shoulong Zhang, Xiaoqiang Liu, Sisi Zhuang, Yuan Zhang, Pengfei Wan, Di Zhang, Shuai Li

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted by ICLR'25

MES-RAG: Bringing Multi-modal, Entity-Storage, and Secure Enhancements to RAG

Pingyu Wu, Daiheng Gao, Jing Tang, Huimin Chen, Wenbo Zhou, Weiming Zhang, Nenghai Yu

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Comments NAACL 2025

MusicLIME: Explainable Multimodal Music Understanding

Theodoros Sotirou, Vassilis Lyberatos, Orfeas Menis Mastromichalakis, Giorgos Stamou

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI、eess.AS

Comments GitHub repository: https://github.com/IamTheo2000/MusicLIME. To be presented at ICASSP 2025