多模态大模型

Learning to Unify Audio, Visual and Text for Audio-Enhanced Multilingual Visual Answer Localization

Zhibin Wen, Bin Li

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CL、cs.AI、cs.MM

Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities

Zhifei Xie, Changqiao Wu

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Technical report, work in progress. Demo and code: https://github.com/gpt-omni/mini-omni2

Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation

Maohao Shen, Shun Zhang, Jilong Wu, Zhiping Xiu, Ehab AlBadawy, Yiting Lu, Mike Seltzer, Qing He

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark

S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, Dinesh Manocha

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Project Website: https://sakshi113.github.io/mmau_homepage/

MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization

Ruiqi Li, Siqi Zheng, Xize Cheng, Ziang Zhang, Shengpeng Ji, Zhou Zhao

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Working in progress

Evaluating the Impact of Warning Modalities and False Alarms in Pedestrian Crossing Alert System

Hesham Alyamani, Yucheng Yang, David Noyce, Madhav Chitturi, Kassem Fawaz

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)

Comments 15 pages

UniMuMo: Unified Text, Music and Motion Generation

Han Yang, Kun Su, Yutong Zhang, Jiaben Chen, Kaizhi Qian, Gaowen Liu, Chuang Gan

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systems

Kentaro Mitsui, Koh Mitsuda, Toshiaki Wakatsuki, Yukiya Hono, Kei Sawada

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 9 pages, 6 figures, 4 tables, accepted for Findings of EMNLP 2024. Demo samples: https://rinnakk.github.io/research/publications/PSLM

WavLLM: Towards Robust and Adaptive Speech Large Language Model

Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, Furu Wei

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments accepted by EMNLP2024 findings

Bridging Paintings and Music -- Exploring Emotion based Music Generation through Paintings

Tanisha Hisariya, Huan Zhang, Jinhua Liang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesis

Qi Yang, Binjie Mao, Zili Wang, Xing Nie, Pengfei Gao, Ying Guo, Cheng Zhen, Pengfei Yan, Shiming Xiang

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 14 pages, 11 figures

Interpretable Convolutional SyncNet

Sungjoon Park, Jaesub Yun, Donggeon Lee, Minsik Park

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 8+5 pages

Foundation Models for Music: A Survey

Yinghao Ma, Anders Øland, Anton Ragni, Bleiz MacSen Del Sette, Charalampos Saitis, Chris Donahue, Chenghua Lin, Christos Plachouras, Emmanouil Benetos, Elona Shatri, Fabio Morreale, Ge Zhang, György Fazekas, Gus Xia, Huan Zhang, Ilaria Manco, Jiawen Huang, Julien Guinot, Liwei Lin, Luca Marinelli, Max W. Y. Lam, Megha Sharma, Qiuqiang Kong, Roger B. Dannenberg, Ruibin Yuan, Shangda Wu, Shih-Lun Wu, Shuqi Dai, Shun Lei, Shiyin Kang, Simon Dixon, Wenhu Chen, Wenhao Huang, Xingjian Du, Xingwei Qu, Xu Tan, Yizhi Li, Zeyue Tian, Zhiyong Wu, Zhizheng Wu, Ziyang Ma, Ziyu Wang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Bidirectional Intent Communication: A Role for Large Foundation Models

Tim Schreiter, Rishi Hazra, Jens Rüppel, Andrey Rudenko

专题命中 音频语音多模态 ：multimodal(abstract)；multimodal foundation model(abstract)

Comments 2024 33rd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), Workshop: Large Language Models in the RoMan Age

ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding

Zhen Chen, Zongming Zhang, Wenwu Guo, Xingjian Luo, Long Bai, Jinlin Wu, Hongliang Ren, Hongbin Liu

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments This work is accepted by IROS 2024 (Oral)

Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization

Navonil Majumder, Chia-Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, Soujanya Poria

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted at ACM MM 2024

Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity

Santiago Pascual, Chunghsin Yeh, Ioannis Tsiamas, Joan Serrà

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Accepted to ECCV 2024

Semantic Grouping Network for Audio Source Separation

Shentong Mo, Yapeng Tian

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT

Zhihao Du, Jiaming Wang, Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, Wen Wang, Siqi Zheng, Chang Zhou, Zhijie Yan, Shiliang Zhang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 10 pages, work in progress

Towards Generating Diverse Audio Captions via Adversarial Training

Xinhao Mei, Xubo Liu, Jianyuan Sun, Mark D. Plumbley, Wenwu Wang

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments Accepted to TASLP

Investigating the impact of 2D gesture representation on co-speech gesture generation

Teo Guichoux, Laure Soulier, Nicolas Obin, Catherine Pelachaud

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 8 pages. Paper accepted at WACAI 2024

Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction

Haoqiu Yan, Yongxin Zhu, Kai Zheng, Bing Liu, Haoyu Cao, Deqiang Jiang, Linli Xu

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 9 pages, 3 figures, ACL24 accepted

COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning

Jing Pan, Jian Wu, Yashesh Gaur, Sunit Sivasankaran, Zhuo Chen, Shujie Liu, Jinyu Li

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Language

Mark Hamilton, Andrew Zisserman, John R. Hershey, William T. Freeman

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Computer Vision and Pattern Recognition 2024

AudioSetMix: Enhancing Audio-Language Datasets with LLM-Assisted Augmentations

David Xu

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments typos corrected

LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognition

Sreyan Ghosh, Sonal Kumar, Ashish Seth, Purva Chiniya, Utkarsh Tyagi, Ramani Duraiswami, Dinesh Manocha

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments InterSpeech 2024. Code and Data: https://github.com/Sreyan88/LipGER

Unveiling Hidden Factors: Explainable AI for Feature Boosting in Speech Emotion Recognition

Alaa Nfissi, Wassim Bouachir, Nizar Bouguila, Brian Mishara

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Published in: Springer Nature International Journal of Applied Intelligence (2024)

Journal ref Applied Intelligence (2024), 1-24

Voice Jailbreak Attacks Against GPT-4o

Xinyue Shen, Yixin Wu, Michael Backes, Yang Zhang

专题命中 音频语音多模态 ：multimodal(abstract)；MLLM(abstract)

Faces that Speak: Jointly Synthesising Talking Face and Speech from Text

Youngjoon Jang, Ji-Hoon Kim, Junseok Ahn, Doyeop Kwak, Hong-Sun Yang, Yoon-Cheol Ju, Il-Hwan Kim, Byeong-Yeol Kim, Joon Son Chung

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments CVPR 2024

Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models

Yuchen Hu, Chen Chen, Chengwei Qin, Qiushi Zhu, Eng Siong Chng, Ruizhe Li

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 14 pages, Accepted by ACL 2024