多模态大模型

I see what you mean: Co-Speech Gestures for Reference Resolution in Multimodal Dialogue

Esam Ghaleb, Bulat Khaertdinov, Aslı Özyürek, Raquel Fernández

机构 * Multimodal Language Department, Max Planck Institute for Psycholinguistics（马克斯·普朗克心理语言学研究所多模态语言部门） ； Donders Institute for Brain, Cognition and Behaviour, Radboud University（拉德堡德大学大脑、认知与行为研究所） ； Department of Advanced Computing Sciences, Maastricht University（马斯特里赫特大学高级计算科学系） ； Institute for Logic, Language and Computation, University of Amsterdam（阿姆斯特丹大学逻辑、语言与计算研究所） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted to Findings of ACL 2025

ASR-enhanced Multimodal Representation Learning for Cross-Domain Product Retrieval

Ruixiang Zhao, Jian Jia, Yan Li, Xuehan Bai, Quan Chen, Han Li, Peng Jiang, Xirong Li

机构 * Renmin University of China（中国人民大学） ； Kuaishou Technology（快手科技） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments accepted for publication as a REGULAR paper in the IEEE Transactions on Multimedia

PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decoding

Jiajun He, Tomoki Toda

机构 * Graduate School of Informatics, Nagoya University（信息学研究科，名古屋大学） ； Information Technology Center, Nagoya University（信息科技中心，名古屋大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted by IEEE TASLP 2025

Towards disentangling the contributions of articulation and acoustics in multimodal phoneme recognition

Sean Foley, Hong Nguyen, Jihwan Lee, Sudarsana Reddy Kadiri, Dani Byrd, Louis Goldstein, Shrikanth Narayanan

机构 * University of Southern CaliforniaUSA（美国南加州大学） ； Signal Analysis and Interpretation Laboratory（信号分析与解释实验室） ； Department of Linguistics（语言学系） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)

Audio-Visual Contact Classification for Tree Structures in Agriculture

Ryan Spears, Moonyoung Lee, George Kantor, Oliver Kroemer

机构 * Carnegie Mellon University（卡内基梅隆大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)

Comments 8 pages

Incorporating brain-inspired mechanisms for multimodal learning in artificial intelligence

Xiang He, Dongcheng Zhao, Yang Li, Qingqun Kong, Xin Yang, Yi Zeng

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)

Comments The manuscript is under review and the code is available at https://github.com/Brain-Cog-Lab/IEMF

Audio-Visual Class-Incremental Learning for Fish Feeding intensity Assessment in Aquaculture

Meng Cui, Xianghu Yue, Xinyuan Qian, Jinzheng Zhao, Haohe Liu, Xubo Liu, Daoliang Li, Wenwu Wang

机构 * Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey（视觉、语音和信号处理中心（CVSSP）， Surrey大学） ； College of Intelligence and Computing, Tianjin University（智能与计算学院，天津大学） ； School of Computer and Communication Engineering, University of Science and Technology Beijing（计算机与通信工程学院，北京科技大学） ； National Innovation Center for Digital Fishery, China Agricultural University（数字渔业国家创新中心，中国农业大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)

AMPS: ASR with Multimodal Paraphrase Supervision

Abhishek Gupta, Amruta Parulekar, Sameep Chattopadhyay, Preethi Jyothi

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis

Pan Wang, Qiang Zhou, Yawen Wu, Tianlong Chen, Jingtong Hu

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments AAAI 2025 accepted

AV-PedAware: Self-Supervised Audio-Visual Fusion for Dynamic Pedestrian Awareness

Yizhuo Yang, Shenghai Yuan, Muqing Cao, Jianfei Yang, Lihua Xie

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)

Comments This work has been accepted for publication at the 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Personal use is permitted. For other uses, permission from IEEE is required

Video-Guided Foley Sound Generation with Multimodal Controls

Ziyang Chen, Prem Seetharaman, Bryan Russell, Oriol Nieto, David Bourgin, Andrew Owens, Justin Salamon

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted at CVPR 2025. Project site: https://ificl.github.io/MultiFoley/

LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport

Kyeongha Rho, Hyeongkeun Lee, Valentio Iverson, Joon Son Chung

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 5 pages, 2 figures; Accepted to ICASSP 2025

Sequential Contrastive Audio-Visual Learning

Ioannis Tsiamas, Santiago Pascual, Chunghsin Yeh, Joan Serrà

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments ICASSP 2025. Version 1 contains more details

Audio-Visual Deepfake Detection With Local Temporal Inconsistencies

Marcella Astrid, Enjie Ghorbel, Djamila Aouada

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted in ICASSP 2025

Do Audio-Visual Segmentation Models Truly Segment Sounding Objects?

Jia Li, Wenjie Zhao, Ziru Huang, Yunhui Guo, Yapeng Tian

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.AI、cs.MM、eess.AS

Audio-visual cross-modality knowledge transfer for machine learning-based in-situ monitoring in laser additive manufacturing

Jiarui Xie, Mutahar Safdar, Lequn Chen, Seung Ki Moon, Yaoyao Fiona Zhao

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)

Comments 47 pages, 19 figures, 6 tables

Progressive Confident Masking Attention Network for Audio-Visual Segmentation

Yuxuan Wang, Jinchao Zhu, Feng Dong, Shuyue Zhu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 23 pages, 11 figures, submitted to Elsevier Knowledge-Based System

Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment

Joanna Hong, Sanjeel Parekh, Honglie Chen, Jacob Donley, Ke Tan, Buye Xu, Anurag Kumar

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity

Weixin Liang, Junhong Shen, Genghan Zhang, Ning Dong, Luke Zettlemoyer, Lili Yu

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speech

Rui Liu, Shuwei He, Yifan Hu, Haizhou Li

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 9 pages,2 figures, Accepted by AAAI'2025

Diffusion-based Unsupervised Audio-visual Speech Enhancement

Jean-Eudes Ayilo, Mostafa Sadeghi, Romain Serizel, Xavier Alameda-Pineda

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.AI、eess.AS

Journal ref International Conference on Acoustics Speech and Signal Processing (ICASSP), IEEE, Apr 2025, Hyderabad, India

MinMo: A Multimodal Large Language Model for Seamless Voice Interaction

Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, Yabin Li, Xiang Lv, Jiaqing Liu, Haoneng Luo, Bin Ma, Chongjia Ni, Xian Shi, Jialong Tang, Hui Wang, Hao Wang, Wen Wang, Yuxuan Wang, Yunlan Xu, Fan Yu, Zhijie Yan, Yexin Yang, Baosong Yang, Xian Yang, Guanrou Yang, Tianyu Zhao, Qinglin Zhang, Shiliang Zhang, Nan Zhao, Pei Zhang, Chong Zhang, Jinren Zhou

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Work in progress. Authors are listed in alphabetical order by family name

AV-DTEC: Self-Supervised Audio-Visual Fusion for Drone Trajectory Estimation and Classification

Zhenyuan Xiao, Yizhuo Yang, Guili Xu, Xianglong Zeng, Shenghai Yuan

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Submitted to ICRA 2025

Data-Centric Improvements for Enhancing Multi-Modal Understanding in Spoken Conversation Modeling

Maximillian Chen, Ruoxi Sun, Sercan Ö. Arık

专题命中 音频语音多模态 ：multi-modal(title)；multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 22 pages, 6 figures, 14 tables

Discrepancy-Aware Attention Network for Enhanced Audio-Visual Zero-Shot Learning

RunLin Yu, Yipu Gong, Wenrui Li, Aiwen Sun, Mengren Zheng

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Enriching Multimodal Sentiment Analysis through Textual Emotional Descriptions of Visual-Audio Content

Sheng Wu, Xiaobao Wang, Longbiao Wang, Dongxiao He, Jianwu Dang

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI、eess.AS

Journal ref AAAI 2025

Collaborative Hybrid Propagator for Temporal Misalignment in Audio-Visual Segmentation

Kexin Li, Zongxin Yang, Yi Yang, Jun Xiao

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Personalized Multimodal Large Language Models: A Survey

Junda Wu, Hanjia Lyu, Yu Xia, Zhehao Zhang, Joe Barrow, Ishita Kumar, Mehrnoosh Mirtaheri, Hongjie Chen, Ryan A. Rossi, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, Jiuxiang Gu, Nesreen K. Ahmed, Yu Wang, Xiang Chen, Hanieh Deilamsalehy, Namyong Park, Sungchul Kim, Huanrui Yang, Subrata Mitra, Zhengmian Hu, Nedim Lipka, Dang Nguyen, Yue Zhao, Jiebo Luo, Julian McAuley

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Measuring Sound Symbolism in Audio-visual Models

Wei-Cheng Tseng, Yi-Jen Shih, David Harwath, Raymond Mooney

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments SLT 2024

PIAST: A Multimodal Piano Dataset with Audio, Symbolic and Text

Hayeon Bang, Eunjin Choi, Megan Finch, Seungheon Doh, Seolhee Lee, Gyeong-Hoon Lee, Juhan Nam

专题命中 音频语音多模态 ：multimodal(title)；multi-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments Accepted for publication at the 3rd Workshop on NLP for Music and Audio (NLP4MusA 2024)