多模态大模型

Vision-to-Music Generation: A Survey

Zhaokai Wang, Chenxi Bao, Le Zhuo, Jingrui Han, Yang Yue, Yihong Tang, Victor Shea-Jay Huang, Yue Liao

机构 * Shanghai Jiao Tong University（上海交通大学） ； Music Tech Lab, DynamiX（音乐科技实验室，DynamiX） ； Shanghai AI Laboratory（上海人工智能实验室） ； Beijing Film Academy（北京电影学院） ； Tsinghua University（清华大学） ； McGill University（麦吉尔大学） ； The Chinese University of Hong Kong（香港中文大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Journal ref ISMIR 2025 "A Survey on Vision to Music Generation: Methods, Datasets, Evaluation, and Challenges"

Developing an AI-Guided Assistant Device for the Deaf and Hearing Impaired

Jiayu, Liu

机构 * Jiayu (Jerry) Liu（独立研究者） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 eess.AS

On the development of an AI performance and behavioural measures for teaching and classroom management

Andreea I. Niculescu, Jochen Ehnes, Chen Yi, Du Jiawei, Tay Chiat Pin, Joey Tianyi Zhou, Vigneshwaran Subbaraju, Teh Kah Kuan, Tran Huy Dat, John Komar, Gi Soong Chee, Kenneth Kwok

机构 * A*STAR Inst. for Infocomm Research（A*STAR信息与通信研究所） ； National Institute of Education NTU（国家教育研究所NTU） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments 7 pages, 10 figures, A video demonstration of the teacher trainer dashboard can be accessed here: https://vimeo.com/1076482827

Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation

Yingshan Liang, Keyu Fan, Zhicheng Du, Yiran Wang, Qingyang Shi, Xinyu Zhang, Jiasheng Lu, Peiwu Qin

机构 * Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院） ； Huawei Technologies Co., Ltd.（华为技术有限公司） 纠错

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.AI、cs.MM

CLEP-DG: Contrastive Learning for Speech Emotion Domain Generalization via Soft Prompt Tuning

Jiacheng Shi, Yanfu Zhang, Ye Gao

机构 * Department of Computer Science（计算机科学系） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments Accepted to Interspeech2025

An Audio-centric Multi-task Learning Framework for Streaming Ads Targeting on Spotify

Shivam Verma, Vivian Chen, Darren Mei

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments Accepted at KDD 2025

Harmony-Aware Music-driven Motion Synthesis with Perceptual Constraint on UGC Datasets

Xinyi Wu, Haohong Wang, Aggelos K. Katsaggelos

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.MM

TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages

Minsu Kim, Jee-weon Jung, Hyeongseop Rha, Soumi Maiti, Siddhant Arora, Xuankai Chang, Shinji Watanabe, Yong Man Ro

机构 * Integrated Vision and Language Lab., School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST)（整合视觉与语言实验室，电气工程学院，韩国科学技术院（KAIST）） ； Language Technologies Institute, Carnegie Mellon University（语言技术研究所，卡内基梅隆大学） 纠错

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments IEEE TMM

Improving Speech Emotion Recognition Through Cross Modal Attention Alignment and Balanced Stacking Model

Lucas Ueda, João Lima, Leonardo Marques, Paula Costa

机构 * Ueda Lima Marques Costa Universidade Estadual de Campinas (UNICAMP)Brazil CPQDBrazil（乌埃达、利玛、马尔奎斯、科斯塔 马丁斯大学（UNICAMP）巴西 CPQD 巴西） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments Accepted by INTERSPEECH 2025

Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration

Hao Zhong, Muzhi Zhu, Zongze Du, Zheng Huang, Canyu Zhao, Mingyu Liu, Wen Wang, Hao Chen, Chunhua Shen

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments Project page: https://aim-uofa.github.io/OmniR1

VersatileMotion: A Unified Framework for Motion Synthesis and Comprehension

Zeyu Ling, Bo Han, Shiyang Li, Jikang Cheng, Hongdeng Shen, Changqing Zou

机构 * Zhejiang University（浙江大学） ； ByteDance（字节跳动） ； University of the Chinese Academy（中国科学院大学） ； Zhejiang Lab（浙江实验室） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages

Shangda Wu, Zhancheng Guo, Ruibin Yuan, Junyan Jiang, Seungheon Doh, Gus Xia, Juhan Nam, Xiaobing Li, Feng Yu, Maosong Sun

机构 * Page 9（页面9） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments 20 pages, 8 figures, 12 tables, accepted by ACL 2025

How Does Audio Influence Visual Attention in Omnidirectional Videos? Database and Model

Yuxin Zhu, Huiyu Duan, Kaiwei Zhang, Yucheng Zhu, Xilei Zhu, Long Teng, Xiongkuo Min, Guangtao Zhai

机构 * Shanghai Jiao Tong University（上海交通大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV

Real-Time Auralization for First-Person Vocal Interaction in Immersive Virtual Environments

Mauricio Flores-Vargas, Enda Bates, Rachel McDonnell

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 eess.AS

Unifying EEG and Speech for Emotion Recognition: A Two-Step Joint Learning Framework for Handling Missing EEG Data During Inference

Upasana Tiwari, Rupayan Chakraborty, Sunil Kumar Kopparapu

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.AI

Comments 10 pages, 5 figures

Vision-Speech Models: Teaching Speech Models to Converse about Images

Amélie Royer, Moritz Böhle, Gabriel de Marmiesse, Laurent Mazaré, Neil Zeghidour, Alexandre Défossez, Patrick Pérez

专题命中 音频语音多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV

Solution for 8th Competition on Affective & Behavior Analysis in-the-wild

Jun Yu, Yunxiang Zhang, Xilong Lu, Yang Zheng, Yongqi Wang, Lingsi Zhu

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV

NVP-HRI: Zero Shot Natural Voice and Posture-based Human-Robot Interaction via Large Language Model

Yuzhi Lai, Shenghai Yuan, Youssef Nassar, Mingyu Fan, Thomas Weber, Matthias Rätsch

专题命中 音频语音多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.AI

Comments This work has been accepted for publication in ESWA @ 2025 Elsevier. Personal use of this material is permitted. Permission from Elsevier must be obtained for all other uses, including reprinting/redistribution, creating new works, or reuse of any copyrighted components of this work in other media

Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning

Mengshi Qi, Changsheng Lv, Huadong Ma

专题命中 音频语音多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV

Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding

Joshua Jones, Oier Mees, Carmelo Sferrazza, Kyle Stachowicz, Pieter Abbeel, Sergey Levine

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.AI

From MLP to NeoMLP: Leveraging Self-Attention for Neural Fields

Miltiadis Kofinas, Samuele Papa, Efstratios Gavves

专题命中 音频语音多模态 ：multi-modal(abstract)；audio-visual(abstract)；分类 cs.AI

Comments Preprint. Source code: https://github.com/mkofinas/neomlp

OpenMU: Your Swiss Army Knife for Music Understanding

Mengjie Zhao, Zhi Zhong, Zhuoyuan Mao, Shiqi Yang, Wei-Hsiang Liao, Shusuke Takahashi, Hiromi Wakaki, Yuki Mitsufuji

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Resources: https://github.com/sony/openmu

Scaling Wearable Foundation Models

Girish Narayanswamy, Xin Liu, Kumar Ayush, Yuzhe Yang, Xuhai Xu, Shun Liao, Jake Garrison, Shyam Tailor, Jake Sunshine, Yun Liu, Tim Althoff, Shrikanth Narayanan, Pushmeet Kohli, Jiening Zhan, Mark Malhotra, Shwetak Patel, Samy Abdel-Ghaffar, Daniel McDuff

专题命中 音频语音多模态 ：multimodal(abstract)；multimodal foundation model(abstract)；分类 cs.AI

Strong Alone, Stronger Together: Synergizing Modality-Binding Foundation Models with Optimal Transport for Non-Verbal Emotion Recognition

Orchid Chetia Phukan, Mohd Mujtaba Akhtar, Girish, Swarup Ranjan Behera, Sishir Kalita, Arun Balaji Buduru, Rajesh Sharma, S. R Mahadeva Prasanna

专题命中 音频语音多模态 ：multimodal(abstract)；multimodal foundation model(abstract)；分类 eess.AS

Comments Submitted to ICASSP 2025

KMTalk: Speech-Driven 3D Facial Animation with Key Motion Embedding

Zhihao Xu, Shengjie Gong, Jiapeng Tang, Lingyu Liang, Yining Huang, Haojie Li, Shuangping Huang

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments Accepted by ECCV 2024

S^3D-NeRF: Single-Shot Speech-Driven Neural Radiance Field for High Fidelity Talking Head Synthesis

Dongze Li, Kang Zhao, Wei Wang, Yifeng Ma, Bo Peng, Yingya Zhang, Jing Dong

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments ECCV 2024

Advancing Multi-grained Alignment for Contrastive Language-Audio Pre-training

Yiming Li, Zhifang Guo, Xiangdong Wang, Hong Liu

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments ACM MM 2024 (Oral)

RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Network

Xiaozhong Ji, Chuming Lin, Zhonggan Ding, Ying Tai, Junwei Zhu, Xiaobin Hu, Donghao Luo, Yanhao Ge, Chengjie Wang

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Unveiling the Safety of GPT-4o: An Empirical Study using Jailbreak Attacks

Zonghao Ying, Aishan Liu, Xianglong Liu, Dacheng Tao

专题命中 音频语音多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV

LARP: Language Audio Relational Pre-training for Cold-Start Playlist Continuation

Rebecca Salganik, Xiaohao Liu, Yunshan Ma, Jian Kang, Tat-Seng Chua

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 eess.AS