多模态大模型

The Sound of Water: Inferring Physical Properties from Pouring Liquids

Piyush Bagad, Makarand Tapaswi, Cees G. M. Snoek, Andrew Zisserman

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Project page at https://bpiyush.github.io/pouring-water-website. Short version accepted to ICASSP 2025

Read, Watch and Scream! Sound Generation from Text and Video

Yujin Jeong, Yunji Kim, Sanghyuk Chun, Jiyoung Lee

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments AAAI2025, Project page: https://naver-ai.github.io/rewas

PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis

Yifan Xie, Tao Feng, Xin Zhang, Xiangyang Luo, Zixuan Guo, Weijiang Yu, Heng Chang, Fei Ma, Fei Richard Yu

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 9 pages, accepted by AAAI 2025

Separate Anything You Describe

Xubo Liu, Qiuqiang Kong, Yan Zhao, Haohe Liu, Yi Yuan, Yuzhuo Liu, Rui Xia, Yuxuan Wang, Mark D. Plumbley, Wenwu Wang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments Code, benchmark and pre-trained models: AGI/AudioSep" target="_blank" rel="noopener">https://github.com/Audio-AGI/AudioSep

GaussianSpeech: Audio-Driven Gaussian Avatars

Shivangi Aneja, Artem Sevastopolsky, Tobias Kirschstein, Justus Thies, Angela Dai, Matthias Nießner

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Paper Video: https://youtu.be/2VqYoFlYcwQ Project Page: https://shivangi-aneja.github.io/projects/gaussianspeech

SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation

Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, Chao Zhang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Technical report

Generative AI for Music and Audio

Hao-Wen Dong

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments PhD Dissertation

Understanding Audiovisual Deepfake Detection: Techniques, Challenges, Human Factors and Perceptual Insights

Ammarah Hashmi, Sahibzada Adil Shahzad, Chia-Wen Lin, Yu Tsao, Hsin-Min Wang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Learning to Unify Audio, Visual and Text for Audio-Enhanced Multilingual Visual Answer Localization

Zhibin Wen, Bin Li

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CL、cs.AI、cs.MM

Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities

Zhifei Xie, Changqiao Wu

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Technical report, work in progress. Demo and code: https://github.com/gpt-omni/mini-omni2

Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation

Maohao Shen, Shun Zhang, Jilong Wu, Zhiping Xiu, Ehab AlBadawy, Yiting Lu, Mike Seltzer, Qing He

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark

S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, Dinesh Manocha

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Project Website: https://sakshi113.github.io/mmau_homepage/

MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization

Ruiqi Li, Siqi Zheng, Xize Cheng, Ziang Zhang, Shengpeng Ji, Zhou Zhao

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Working in progress

Evaluating the Impact of Warning Modalities and False Alarms in Pedestrian Crossing Alert System

Hesham Alyamani, Yucheng Yang, David Noyce, Madhav Chitturi, Kassem Fawaz

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)

Comments 15 pages

UniMuMo: Unified Text, Music and Motion Generation

Han Yang, Kun Su, Yutong Zhang, Jiaben Chen, Kaizhi Qian, Gaowen Liu, Chuang Gan

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systems

Kentaro Mitsui, Koh Mitsuda, Toshiaki Wakatsuki, Yukiya Hono, Kei Sawada

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 9 pages, 6 figures, 4 tables, accepted for Findings of EMNLP 2024. Demo samples: https://rinnakk.github.io/research/publications/PSLM

WavLLM: Towards Robust and Adaptive Speech Large Language Model

Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, Furu Wei

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments accepted by EMNLP2024 findings

Bridging Paintings and Music -- Exploring Emotion based Music Generation through Paintings

Tanisha Hisariya, Huan Zhang, Jinhua Liang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesis

Qi Yang, Binjie Mao, Zili Wang, Xing Nie, Pengfei Gao, Ying Guo, Cheng Zhen, Pengfei Yan, Shiming Xiang

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 14 pages, 11 figures

Interpretable Convolutional SyncNet

Sungjoon Park, Jaesub Yun, Donggeon Lee, Minsik Park

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 8+5 pages

Foundation Models for Music: A Survey

Yinghao Ma, Anders Øland, Anton Ragni, Bleiz MacSen Del Sette, Charalampos Saitis, Chris Donahue, Chenghua Lin, Christos Plachouras, Emmanouil Benetos, Elona Shatri, Fabio Morreale, Ge Zhang, György Fazekas, Gus Xia, Huan Zhang, Ilaria Manco, Jiawen Huang, Julien Guinot, Liwei Lin, Luca Marinelli, Max W. Y. Lam, Megha Sharma, Qiuqiang Kong, Roger B. Dannenberg, Ruibin Yuan, Shangda Wu, Shih-Lun Wu, Shuqi Dai, Shun Lei, Shiyin Kang, Simon Dixon, Wenhu Chen, Wenhao Huang, Xingjian Du, Xingwei Qu, Xu Tan, Yizhi Li, Zeyue Tian, Zhiyong Wu, Zhizheng Wu, Ziyang Ma, Ziyu Wang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Bidirectional Intent Communication: A Role for Large Foundation Models

Tim Schreiter, Rishi Hazra, Jens Rüppel, Andrey Rudenko

专题命中 音频语音多模态 ：multimodal(abstract)；multimodal foundation model(abstract)

Comments 2024 33rd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), Workshop: Large Language Models in the RoMan Age

ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding

Zhen Chen, Zongming Zhang, Wenwu Guo, Xingjian Luo, Long Bai, Jinlin Wu, Hongliang Ren, Hongbin Liu

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments This work is accepted by IROS 2024 (Oral)

Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization

Navonil Majumder, Chia-Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, Soujanya Poria

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted at ACM MM 2024

Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity

Santiago Pascual, Chunghsin Yeh, Ioannis Tsiamas, Joan Serrà

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Accepted to ECCV 2024

Semantic Grouping Network for Audio Source Separation

Shentong Mo, Yapeng Tian

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT

Zhihao Du, Jiaming Wang, Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, Wen Wang, Siqi Zheng, Chang Zhou, Zhijie Yan, Shiliang Zhang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 10 pages, work in progress

Towards Generating Diverse Audio Captions via Adversarial Training

Xinhao Mei, Xubo Liu, Jianyuan Sun, Mark D. Plumbley, Wenwu Wang

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments Accepted to TASLP

Investigating the impact of 2D gesture representation on co-speech gesture generation

Teo Guichoux, Laure Soulier, Nicolas Obin, Catherine Pelachaud

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 8 pages. Paper accepted at WACAI 2024

Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction

Haoqiu Yan, Yongxin Zhu, Kai Zheng, Bing Liu, Haoyu Cao, Deqiang Jiang, Linli Xu

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 9 pages, 3 figures, ACL24 accepted