多模态大模型 - arXivDaily 专题

2505.20007 2025-06-03 eess.AS cs.SD 70%

Improving Speech Emotion Recognition Through Cross Modal Attention Alignment and Balanced Stacking Model

Lucas Ueda, João Lima, Leonardo Marques, Paula Costa

机构 * Ueda Lima Marques Costa Universidade Estadual de Campinas (UNICAMP)Brazil CPQDBrazil（乌埃达、利玛、马尔奎斯、科斯塔马丁斯大学（UNICAMP）巴西 CPQD 巴西）

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments Accepted by INTERSPEECH 2025

2505.20256 2025-05-27 cs.CV 70%

Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration

Hao Zhong, Muzhi Zhu, Zongze Du, Zheng Huang, Canyu Zhao, Mingyu Liu, Wen Wang, Hao Chen, Chunhua Shen

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments Project page: https://aim-uofa.github.io/OmniR1

2411.17335 2025-05-27 cs.CV 70%

VersatileMotion: A Unified Framework for Motion Synthesis and Comprehension

Zeyu Ling, Bo Han, Shiyang Li, Jikang Cheng, Hongdeng Shen, Changqing Zou

机构 * Zhejiang University（浙江大学）； ByteDance（字节跳动）； University of the Chinese Academy（中国科学院大学）； Zhejiang Lab（浙江实验室）

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

2502.10362 2025-05-20 cs.SD eess.AS 70%

CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages

Shangda Wu, Zhancheng Guo, Ruibin Yuan, Junyan Jiang, Seungheon Doh, Gus Xia, Juhan Nam, Xiaobing Li, Feng Yu, Maosong Sun

机构 * Page 9（页面9）

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments 20 pages, 8 figures, 12 tables, accepted by ACL 2025

2408.05411 2025-05-06 cs.CV 70%

How Does Audio Influence Visual Attention in Omnidirectional Videos? Database and Model

Yuxin Zhu, Huiyu Duan, Kaiwei Zhang, Yucheng Zhu, Xilei Zhu, Long Teng, Xiongkuo Min, Guangtao Zhai

机构 * Shanghai Jiao Tong University（上海交通大学）

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV

2504.04075 2025-04-08 eess.AS cs.HC eess.SP 70%

Real-Time Auralization for First-Person Vocal Interaction in Immersive Virtual Environments

Mauricio Flores-Vargas, Enda Bates, Rachel McDonnell

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 eess.AS

2503.18964 2025-03-26 cs.SD cs.AI 70%

Unifying EEG and Speech for Emotion Recognition: A Two-Step Joint Learning Framework for Handling Missing EEG Data During Inference

Upasana Tiwari, Rupayan Chakraborty, Sunil Kumar Kopparapu

机构 * TCS（塔塔咨询服务公司）

专题命中音频语音多模态：multi-modal(abstract)；cross-modal(abstract)；分类 cs.AI

Comments 10 pages, 5 figures

2503.15633 2025-03-21 cs.CV 70%

Vision-Speech Models: Teaching Speech Models to Converse about Images

Amélie Royer, Moritz Böhle, Gabriel de Marmiesse, Laurent Mazaré, Neil Zeghidour, Alexandre Défossez, Patrick Pérez

机构 * Kyutai

专题命中音频语音多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV

2503.11115 2025-03-17 cs.CV 70%

Solution for 8th Competition on Affective & Behavior Analysis in-the-wild

Jun Yu, Yunxiang Zhang, Xilong Lu, Yang Zheng, Yongqi Wang, Lingsi Zhu

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV

2503.09335 2025-03-13 cs.RO cs.AI 70%

NVP-HRI: Zero Shot Natural Voice and Posture-based Human-Robot Interaction via Large Language Model

Yuzhi Lai, Shenghai Yuan, Youssef Nassar, Mingyu Fan, Thomas Weber, Matthias Rätsch

机构 * University Reutlingen（罗伊特林根大学）； Nanyang Technological University（南洋理工大学）； Donghua University（东华大学）

专题命中音频语音多模态：multimodal(abstract)；multi-modal(abstract)；分类 cs.AI

Comments This work has been accepted for publication in ESWA @ 2025 Elsevier. Personal use of this material is permitted. Permission from Elsevier must be obtained for all other uses, including reprinting/redistribution, creating new works, or reuse of any copyrighted components of this work in other media

2502.12425 2025-02-19 cs.CV 70%

Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning

Mengshi Qi, Changsheng Lv, Huadong Ma

机构 * Beijing University of Posts and Telecommunications（北京邮电大学）

专题命中音频语音多模态：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV

2501.04693 2025-01-16 cs.RO cs.AI 70%

Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding

Joshua Jones, Oier Mees, Carmelo Sferrazza, Kyle Stachowicz, Pieter Abbeel, Sergey Levine

机构 * Berkeley AI Research (BAIR)（伯克利人工智能研究院（BAIR））； UC Berkeley（加州大学伯克利分校）

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.AI

2412.08731 2024-12-13 cs.LG cs.AI stat.ML 70%

From MLP to NeoMLP: Leveraging Self-Attention for Neural Fields

Miltiadis Kofinas, Samuele Papa, Efstratios Gavves

专题命中音频语音多模态：multi-modal(abstract)；audio-visual(abstract)；分类 cs.AI

Comments Preprint. Source code: https://github.com/mkofinas/neomlp

2410.15573 2024-11-28 cs.SD cs.AI cs.CL cs.MM eess.AS 70%

OpenMU: Your Swiss Army Knife for Music Understanding

Mengjie Zhao, Zhi Zhong, Zhuoyuan Mao, Shiqi Yang, Wei-Hsiang Liao, Shusuke Takahashi, Hiromi Wakaki, Yuki Mitsufuji

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Resources: https://github.com/sony/openmu

2410.13638 2024-10-18 cs.LG cs.AI cs.HC 70%

Scaling Wearable Foundation Models

Girish Narayanswamy, Xin Liu, Kumar Ayush, Yuzhe Yang, Xuhai Xu, Shun Liao, Jake Garrison, Shyam Tailor, Jake Sunshine, Yun Liu, Tim Althoff, Shrikanth Narayanan, Pushmeet Kohli, Jiening Zhan, Mark Malhotra, Shwetak Patel, Samy Abdel-Ghaffar, Daniel McDuff

专题命中音频语音多模态：multimodal(abstract)；multimodal foundation model(abstract)；分类 cs.AI

2409.14221 2024-09-24 eess.AS cs.SD 70%

Strong Alone, Stronger Together: Synergizing Modality-Binding Foundation Models with Optimal Transport for Non-Verbal Emotion Recognition

Orchid Chetia Phukan, Mohd Mujtaba Akhtar, Girish, Swarup Ranjan Behera, Sishir Kalita, Arun Balaji Buduru, Rajesh Sharma, S. R Mahadeva Prasanna

专题命中音频语音多模态：multimodal(abstract)；multimodal foundation model(abstract)；分类 eess.AS

Comments Submitted to ICASSP 2025

2409.01113 2024-09-04 cs.CV 70%

KMTalk: Speech-Driven 3D Facial Animation with Key Motion Embedding

Zhihao Xu, Shengjie Gong, Jiapeng Tang, Lingyu Liang, Yining Huang, Haojie Li, Shuangping Huang

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments Accepted by ECCV 2024

2408.09347 2024-08-20 cs.CV 70%

S^3D-NeRF: Single-Shot Speech-Driven Neural Radiance Field for High Fidelity Talking Head Synthesis

Dongze Li, Kang Zhao, Wei Wang, Yifeng Ma, Bo Peng, Yingya Zhang, Jing Dong

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments ECCV 2024

2408.07919 2024-08-16 eess.AS 70%

Advancing Multi-grained Alignment for Contrastive Language-Audio Pre-training

Yiming Li, Zhifang Guo, Xiangdong Wang, Hong Liu

专题命中音频语音多模态：multi-modal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments ACM MM 2024 (Oral)

2406.18284 2024-08-09 cs.CV 70%

RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Network

Xiaozhong Ji, Chuming Lin, Zhonggan Ding, Ying Tai, Junwei Zhu, Xiaobin Hu, Donghao Luo, Yanhao Ge, Chengjie Wang

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

2406.06302 2024-07-04 cs.CR cs.CV 70%

Unveiling the Safety of GPT-4o: An Empirical Study using Jailbreak Attacks

Zonghao Ying, Aishan Liu, Xianglong Liu, Dacheng Tao

专题命中音频语音多模态：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV

2406.14333 2024-06-21 cs.IR cs.SD eess.AS 70%

LARP: Language Audio Relational Pre-training for Cold-Start Playlist Continuation

Rebecca Salganik, Xiaohao Liu, Yunshan Ma, Jian Kang, Tat-Seng Chua

专题命中音频语音多模态：multi-modal(abstract)；cross-modal(abstract)；分类 eess.AS

2406.10056 2024-06-17 cs.SD eess.AS 70%

UniAudio 1.5: Large Language Model-driven Audio Codec is A Few-shot Audio Task Learner

Dongchao Yang, Haohan Guo, Yuanyuan Wang, Rongjie Huang, Xiang Li, Xu Tan, Xixin Wu, Helen Meng

专题命中音频语音多模态：multi-modal(abstract)；cross-modal(abstract)；分类 eess.AS

2401.06395 2024-04-05 cs.CV 70%

ModaVerse: Efficiently Transforming Modalities with LLMs

Xinyu Wang, Bohan Zhuang, Qi Wu

专题命中音频语音多模态：multi-modal(abstract)；MLLM(abstract)；分类 cs.CV

Comments CVPR2024

2309.08730 2024-04-03 eess.AS cs.AI cs.CL cs.MM cs.SD 70%

MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response

Zihao Deng, Yinghao Ma, Yudong Liu, Rongchen Guo, Ge Zhang, Wenhu Chen, Wenhao Huang, Emmanouil Benetos

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI、cs.MM

Journal ref 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics

2403.07938 2024-03-14 cs.SD cs.AI cs.CV cs.LG cs.MM eess.AS 70%

Text-to-Audio Generation Synchronized with Videos

Shentong Mo, Jing Shi, Yapeng Tian

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments arXiv admin note: text overlap with arXiv:2305.12903

2401.03697 2024-03-08 cs.SD eess.AS 70%

An audio-quality-based multi-strategy approach for target speaker extraction in the MISP 2023 Challenge

Runduo Han, Xiaopeng Yan, Weiming Xu, Pengcheng Guo, Jiayao Sun, He Wang, Quan Lu, Ning Jiang, Lei Xie

专题命中音频语音多模态：multi-modal(abstract)；audio-visual(abstract)；分类 eess.AS

Comments Accepted by ICASSP 2024

2306.02972 2024-03-08 eess.AS cs.LG 70%

Simultaneous or Sequential Training? How Speech Representations Cooperate in a Multi-Task Self-Supervised Learning System

Khazar Khorrami, María Andrea Cruz Blandón, Tuomas Virtanen, Okko Räsänen

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 eess.AS

Comments 5 pages, accepted by EUSIPCO 2023

2403.01226 2024-03-05 cs.CV 70%

DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction

Junwen Xiong, Peng Zhang, Tao You, Chuanyue Li, Wei Huang, Yufei Zha

专题命中音频语音多模态：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments 15 pages, CVPR24

2402.16153 2024-02-27 cs.SD cs.AI cs.CL cs.LG cs.MM eess.AS 70%

ChatMusician: Understanding and Generating Music Intrinsically with LLM

Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, Ziyang Ma, Liumeng Xue, Ziyu Wang, Qin Liu, Tianyu Zheng, Yizhi Li, Yinghao Ma, Yiming Liang, Xiaowei Chi, Ruibo Liu, Zili Wang, Pengfei Li, Jingcheng Wu, Chenghua Lin, Qifeng Liu, Tao Jiang, Wenhao Huang, Wenhu Chen, Emmanouil Benetos, Jie Fu, Gus Xia, Roger Dannenberg, Wei Xue, Shiyin Kang, Yike Guo

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、cs.AI、cs.MM

Comments GitHub: https://shanghaicannon.github.io/ChatMusician/

1. 音频语音多模态 4597 篇

Improving Speech Emotion Recognition Through Cross Modal Attention Alignment and Balanced Stacking Model

Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration

VersatileMotion: A Unified Framework for Motion Synthesis and Comprehension

CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages

How Does Audio Influence Visual Attention in Omnidirectional Videos? Database and Model

Real-Time Auralization for First-Person Vocal Interaction in Immersive Virtual Environments

Unifying EEG and Speech for Emotion Recognition: A Two-Step Joint Learning Framework for Handling Missing EEG Data During Inference

Vision-Speech Models: Teaching Speech Models to Converse about Images

Solution for 8th Competition on Affective & Behavior Analysis in-the-wild

NVP-HRI: Zero Shot Natural Voice and Posture-based Human-Robot Interaction via Large Language Model

Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning

Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding

From MLP to NeoMLP: Leveraging Self-Attention for Neural Fields

OpenMU: Your Swiss Army Knife for Music Understanding

Scaling Wearable Foundation Models

Strong Alone, Stronger Together: Synergizing Modality-Binding Foundation Models with Optimal Transport for Non-Verbal Emotion Recognition

KMTalk: Speech-Driven 3D Facial Animation with Key Motion Embedding

S^3D-NeRF: Single-Shot Speech-Driven Neural Radiance Field for High Fidelity Talking Head Synthesis

Advancing Multi-grained Alignment for Contrastive Language-Audio Pre-training

RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Network

Unveiling the Safety of GPT-4o: An Empirical Study using Jailbreak Attacks

LARP: Language Audio Relational Pre-training for Cold-Start Playlist Continuation

UniAudio 1.5: Large Language Model-driven Audio Codec is A Few-shot Audio Task Learner

ModaVerse: Efficiently Transforming Modalities with LLMs

MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response

Text-to-Audio Generation Synchronized with Videos

An audio-quality-based multi-strategy approach for target speaker extraction in the MISP 2023 Challenge

Simultaneous or Sequential Training? How Speech Representations Cooperate in a Multi-Task Self-Supervised Learning System

DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction

ChatMusician: Understanding and Generating Music Intrinsically with LLM