多模态大模型 - arXivDaily 专题

2212.08071 2023-07-18 cs.CV cs.MM cs.SD eess.AS 67%

MAViL: Masked Audio-Video Learners

Po-Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Haoqi Fan, Yanghao Li, Shang-Wen Li, Gargi Ghosh, Jitendra Malik, Christoph Feichtenhofer

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Technical report

2306.12925 2023-06-23 cs.CL cs.AI cs.SD eess.AS stat.ML 67%

AudioPaLM: A Large Language Model That Can Speak and Listen

Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, Hannah Muckenhirn, Dirk Padfield, James Qin, Danny Rozenberg, Tara Sainath, Johan Schalkwyk, Matt Sharifi, Michelle Tadmor Ramanovich, Marco Tagliasacchi, Alexandru Tudor, Mihajlo Velimirović, Damien Vincent, Jiahui Yu, Yongqiang Wang, Vicky Zayats, Neil Zeghidour, Yu Zhang, Zhishuai Zhang, Lukas Zilka, Christian Frank

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Technical report

2209.15329 2023-06-16 cs.CL cs.AI eess.AS 67%

SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data

Ziqiang Zhang, Sanyuan Chen, Long Zhou, Yu Wu, Shuo Ren, Shujie Liu, Zhuoyuan Yao, Xun Gong, Lirong Dai, Jinyu Li, Furu Wei

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments We have corrected the errors in the pre-training data for SpeechLM-P Base models, new results are updated

2306.02596 2023-06-06 eess.AS cs.CL cs.CV 67%

A Novel Interpretable and Generalizable Re-synchronization Model for Cued Speech based on a Multi-Cuer Corpus

Lufei Gao, Shan Huang, Li Liu

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments 5 pages, 4 figures, Accepted to INTERSPEECH2023

2305.17993 2023-05-30 cs.SD cs.AI cs.MM eess.AS 67%

Multi-Scale Attention for Audio Question Answering

Guangyao Li, Yixin Xu, Di Hu

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments Accepted by InterSpeech 2023

2301.13003 2023-05-30 cs.CL cs.AI cs.SD eess.AS 67%

Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation

Minglun Han, Feilong Chen, Jing Shi, Shuang Xu, Bo Xu

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted by INTERSPEECH 2023

2305.17499 2023-05-30 cs.CL cs.MM eess.AS 67%

CIF-PT: Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Training

Linhao Dong, Zhecheng An, Peihao Wu, Jun Zhang, Lu Lu, Zejun Ma

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted by ACL 2023 Findings

2305.13738 2023-05-24 cs.CL cs.AI cs.CV 67%

i-Code Studio: A Configurable and Composable Framework for Integrative AI

Yuwei Fang, Mahmoud Khademi, Chenguang Zhu, Ziyi Yang, Reid Pryzant, Yichong Xu, Yao Qian, Takuya Yoshioka, Lu Yuan, Michael Zeng, Xuedong Huang

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

2304.12995 2023-04-26 cs.CL cs.AI cs.SD eess.AS 67%

AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head

Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, Yi Ren, Zhou Zhao, Shinji Watanabe

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

2304.11093 2023-04-24 cs.CL cs.AI cs.CV cs.LG 67%

Hi Sheldon! Creating Deep Personalized Characters from TV Shows

Meidai Xuanyuan, Yuwang Wang, Honglei Guo, Xiao Ma, Yuchen Guo, Tao Yu, Qionghai Dai

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

2304.08490 2023-04-18 cs.CV cs.MM cs.SD eess.AS 67%

Conditional Generation of Audio from Video via Foley Analogies

Yuexi Du, Ziyang Chen, Justin Salamon, Bryan Russell, Andrew Owens

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments CVPR 2023

2211.08367 2023-04-18 cs.SD cs.CV cs.MM eess.AS 67%

FlowGrad: Using Motion for Visual Sound Source Localization

Rajsuryan Singh, Pablo Zinemanas, Xavier Serra, Juan Pablo Bello, Magdalena Fuentes

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted in ICASSP 2023

2304.00988 2023-04-04 cs.AI cs.MM cs.SD eess.AS 67%

The Music Annotation Pattern

Jacopo de Berardinis, Albert Meroño-Peñuela, Andrea Poltronieri, Valentina Presutti

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 12 pages, 3 figures. Proceedings of the 13th Workshop on Ontology Design and Patterns, edited by V. Svátek et al., WOP, 2022

2303.17200 2023-04-04 cs.CV cs.AI cs.SD eess.AS 67%

SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision

Xubo Liu, Egor Lakomkin, Konstantinos Vougioukas, Pingchuan Ma, Honglie Chen, Ruiming Xie, Morrie Doulaty, Niko Moritz, Jáchym Kolář, Stavros Petridis, Maja Pantic, Christian Fuegen

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments IEEE/CVF CVPR 2023

2303.17517 2023-03-31 cs.CL cs.CV cs.SD eess.AS 67%

Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samples

Hyeonggon Ryu, Arda Senocak, In So Kweon, Joon Son Chung

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments ICASSP 2023

2303.09455 2023-03-17 cs.CL cs.CV cs.LG cs.SD eess.AS 67%

Learning Cross-lingual Visual Speech Representations

Andreas Zinonos, Alexandros Haliassos, Pingchuan Ma, Stavros Petridis, Maja Pantic

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.CL、eess.AS

2303.05007 2023-03-16 cs.CR cs.CV cs.MM cs.SD eess.AS 67%

Towards Robust Image-in-Audio Deep Steganography

Jaume Ros, Margarita Geleta, Jordi Pons, Xavier Giro-i-Nieto

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 8 pages, 5 figures, 2 tables

2302.13321 2023-02-28 cs.SD cs.CL cs.MM eess.AS 67%

Multi-Modality in Music: Predicting Emotion in Music from High-Level Audio Features and Lyrics

Tibor Krols, Yana Nikolova, Ninell Oldenburg

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments 12 pages, incl. 2 pages appendix

2302.08841 2023-02-20 cs.SD cs.CV cs.LG cs.MM eess.AS 67%

Lip-to-Speech Synthesis in the Wild with Multi-task Learning

Minsu Kim, Joanna Hong, Yong Man Ro

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted at ICASSP 2023. Demo available: https://github.com/joannahong/Lip-to-Speech-Synthesis-in-the-Wild

2208.00061 2023-02-17 cs.CV cs.MM cs.SD eess.AS 67%

UAVM: Towards Unifying Audio and Visual Models

Yuan Gong, Alexander H. Liu, Andrew Rouditchenko, James Glass

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Published in Signal Processing Letters. Code at https://github.com/YuanGongND/uavm

Journal ref IEEE Signal Processing Letters, vol. 29, pp. 2437-2441, 2022

2206.08312 2023-01-24 cs.SD cs.CV cs.MM eess.AS 67%

SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning

Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip W Robinson, Kristen Grauman

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Camera-ready version. Website: https://soundspaces.org. Project page: https://vision.cs.utexas.edu/projects/soundspaces2

1509.08973 2023-01-18 cs.AI cs.CL cs.CV cs.RO 67%

Symbol Emergence in Robotics: A Survey

Tadahiro Taniguchi, Takayuki Nagai, Tomoaki Nakamura, Naoto Iwahashi, Tetsuya Ogata, Hideki Asoh

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments submitted to Advanced Robotics

Journal ref Advanced Robotics, 30:11-12, 706-728, 2016

2211.11381 2022-11-22 cs.CV cs.MM cs.SD eess.AS 67%

LISA: Localized Image Stylization with Audio via Implicit Neural Representation

Seung Hyun Lee, Chanyoung Kim, Wonmin Byeon, Sang Ho Yoon, Jinkyu Kim, Sangpil Kim

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

2205.05949 2022-09-28 eess.AS cs.AI cs.MM cs.SD 67%

Automated Audio Captioning: An Overview of Recent Progress and New Challenges

Xinhao Mei, Xubo Liu, Mark D. Plumbley, Wenwu Wang

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments Accepted by EURASIP Journal on Audio Speech and Music Processing

2208.12133 2022-08-26 cs.HC cs.AI cs.MM cs.SD eess.AS 67%

The ReprGesture entry to the GENEA Challenge 2022

Sicheng Yang, Zhiyong Wu, Minglei Li, Mengchen Zhao, Jiuxin Lin, Liyang Chen, Weihong Bao

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 8 pages, 4 figures, ICMI 2022

2208.04498 2022-08-10 cs.CV cs.AI eess.AS eess.IV 67%

Speaker-adaptive Lip Reading with User-dependent Padding

Minsu Kim, Hyunjun Kim, Yong Man Ro

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Accepted at ECCV2022

2203.14250 2022-07-26 cs.CV cs.MM cs.SD eess.AS 67%

End-to-End Active Speaker Detection

Juan Leon Alcazar, Moritz Cordes, Chen Zhao, Bernard Ghanem

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

2207.05692 2022-07-13 cs.MM cs.CL cs.SD eess.AS 67%

Lip-Listening: Mixing Senses to Understand Lips using Cross Modality Knowledge Distillation for Word-Based Models

Hadeel Mabrouk, Omar Abugabal, Nourhan Sakr, Hesham M. Eraqi

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments arXiv admin note: text overlap with arXiv:2108.03543

2206.07684 2022-06-16 cs.CV cs.MM cs.SD eess.AS 67%

AVATAR: Unconstrained Audiovisual Speech Recognition

Valentin Gabeur, Paul Hongsuck Seo, Arsha Nagrani, Chen Sun, Karteek Alahari, Cordelia Schmid

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

2203.02195 2022-06-14 cs.CV cs.AI cs.MM 67%

Voice-Face Homogeneity Tells Deepfake

Harry Cheng, Yangyang Guo, Tianyi Wang, Qi Li, Xiaojun Chang, Liqiang Nie

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 13 pages for peer review. Code will be released at https://github.com/xaCheng1996/VFD

1. 音频语音多模态 4597 篇

MAViL: Masked Audio-Video Learners

AudioPaLM: A Large Language Model That Can Speak and Listen

SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data

A Novel Interpretable and Generalizable Re-synchronization Model for Cued Speech based on a Multi-Cuer Corpus

Multi-Scale Attention for Audio Question Answering

Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation

CIF-PT: Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Training

i-Code Studio: A Configurable and Composable Framework for Integrative AI

AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head

Hi Sheldon! Creating Deep Personalized Characters from TV Shows

Conditional Generation of Audio from Video via Foley Analogies

FlowGrad: Using Motion for Visual Sound Source Localization

The Music Annotation Pattern

SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision

Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samples

Learning Cross-lingual Visual Speech Representations

Towards Robust Image-in-Audio Deep Steganography

Multi-Modality in Music: Predicting Emotion in Music from High-Level Audio Features and Lyrics

Lip-to-Speech Synthesis in the Wild with Multi-task Learning

UAVM: Towards Unifying Audio and Visual Models

SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning

Symbol Emergence in Robotics: A Survey

LISA: Localized Image Stylization with Audio via Implicit Neural Representation

Automated Audio Captioning: An Overview of Recent Progress and New Challenges

The ReprGesture entry to the GENEA Challenge 2022

Speaker-adaptive Lip Reading with User-dependent Padding

End-to-End Active Speaker Detection

Lip-Listening: Mixing Senses to Understand Lips using Cross Modality Knowledge Distillation for Word-Based Models

AVATAR: Unconstrained Audiovisual Speech Recognition

Voice-Face Homogeneity Tells Deepfake