多模态大模型 - arXivDaily 专题

2304.12995 2023-04-26 cs.CL cs.AI cs.SD eess.AS 67%

AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head

Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, Yi Ren, Zhou Zhao, Shinji Watanabe

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

2304.11093 2023-04-24 cs.CL cs.AI cs.CV cs.LG 67%

Hi Sheldon! Creating Deep Personalized Characters from TV Shows

Meidai Xuanyuan, Yuwang Wang, Honglei Guo, Xiao Ma, Yuchen Guo, Tao Yu, Qionghai Dai

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

2304.08490 2023-04-18 cs.CV cs.MM cs.SD eess.AS 67%

Conditional Generation of Audio from Video via Foley Analogies

Yuexi Du, Ziyang Chen, Justin Salamon, Bryan Russell, Andrew Owens

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments CVPR 2023

2211.08367 2023-04-18 cs.SD cs.CV cs.MM eess.AS 67%

FlowGrad: Using Motion for Visual Sound Source Localization

Rajsuryan Singh, Pablo Zinemanas, Xavier Serra, Juan Pablo Bello, Magdalena Fuentes

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted in ICASSP 2023

2304.00988 2023-04-04 cs.AI cs.MM cs.SD eess.AS 67%

The Music Annotation Pattern

Jacopo de Berardinis, Albert Meroño-Peñuela, Andrea Poltronieri, Valentina Presutti

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 12 pages, 3 figures. Proceedings of the 13th Workshop on Ontology Design and Patterns, edited by V. Svátek et al., WOP, 2022

2303.17200 2023-04-04 cs.CV cs.AI cs.SD eess.AS 67%

SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision

Xubo Liu, Egor Lakomkin, Konstantinos Vougioukas, Pingchuan Ma, Honglie Chen, Ruiming Xie, Morrie Doulaty, Niko Moritz, Jáchym Kolář, Stavros Petridis, Maja Pantic, Christian Fuegen

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments IEEE/CVF CVPR 2023

2303.17517 2023-03-31 cs.CL cs.CV cs.SD eess.AS 67%

Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samples

Hyeonggon Ryu, Arda Senocak, In So Kweon, Joon Son Chung

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments ICASSP 2023

2303.09455 2023-03-17 cs.CL cs.CV cs.LG cs.SD eess.AS 67%

Learning Cross-lingual Visual Speech Representations

Andreas Zinonos, Alexandros Haliassos, Pingchuan Ma, Stavros Petridis, Maja Pantic

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.CL、eess.AS

2303.05007 2023-03-16 cs.CR cs.CV cs.MM cs.SD eess.AS 67%

Towards Robust Image-in-Audio Deep Steganography

Jaume Ros, Margarita Geleta, Jordi Pons, Xavier Giro-i-Nieto

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 8 pages, 5 figures, 2 tables

2302.13321 2023-02-28 cs.SD cs.CL cs.MM eess.AS 67%

Multi-Modality in Music: Predicting Emotion in Music from High-Level Audio Features and Lyrics

Tibor Krols, Yana Nikolova, Ninell Oldenburg

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments 12 pages, incl. 2 pages appendix

2302.08841 2023-02-20 cs.SD cs.CV cs.LG cs.MM eess.AS 67%

Lip-to-Speech Synthesis in the Wild with Multi-task Learning

Minsu Kim, Joanna Hong, Yong Man Ro

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted at ICASSP 2023. Demo available: https://github.com/joannahong/Lip-to-Speech-Synthesis-in-the-Wild

2208.00061 2023-02-17 cs.CV cs.MM cs.SD eess.AS 67%

UAVM: Towards Unifying Audio and Visual Models

Yuan Gong, Alexander H. Liu, Andrew Rouditchenko, James Glass

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Published in Signal Processing Letters. Code at https://github.com/YuanGongND/uavm

Journal ref IEEE Signal Processing Letters, vol. 29, pp. 2437-2441, 2022

2206.08312 2023-01-24 cs.SD cs.CV cs.MM eess.AS 67%

SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning

Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip W Robinson, Kristen Grauman

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Camera-ready version. Website: https://soundspaces.org. Project page: https://vision.cs.utexas.edu/projects/soundspaces2

1509.08973 2023-01-18 cs.AI cs.CL cs.CV cs.RO 67%

Symbol Emergence in Robotics: A Survey

Tadahiro Taniguchi, Takayuki Nagai, Tomoaki Nakamura, Naoto Iwahashi, Tetsuya Ogata, Hideki Asoh

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments submitted to Advanced Robotics

Journal ref Advanced Robotics, 30:11-12, 706-728, 2016

2211.11381 2022-11-22 cs.CV cs.MM cs.SD eess.AS 67%

LISA: Localized Image Stylization with Audio via Implicit Neural Representation

Seung Hyun Lee, Chanyoung Kim, Wonmin Byeon, Sang Ho Yoon, Jinkyu Kim, Sangpil Kim

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

2205.05949 2022-09-28 eess.AS cs.AI cs.MM cs.SD 67%

Automated Audio Captioning: An Overview of Recent Progress and New Challenges

Xinhao Mei, Xubo Liu, Mark D. Plumbley, Wenwu Wang

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments Accepted by EURASIP Journal on Audio Speech and Music Processing

2208.12133 2022-08-26 cs.HC cs.AI cs.MM cs.SD eess.AS 67%

The ReprGesture entry to the GENEA Challenge 2022

Sicheng Yang, Zhiyong Wu, Minglei Li, Mengchen Zhao, Jiuxin Lin, Liyang Chen, Weihong Bao

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 8 pages, 4 figures, ICMI 2022

2208.04498 2022-08-10 cs.CV cs.AI eess.AS eess.IV 67%

Speaker-adaptive Lip Reading with User-dependent Padding

Minsu Kim, Hyunjun Kim, Yong Man Ro

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Accepted at ECCV2022

2203.14250 2022-07-26 cs.CV cs.MM cs.SD eess.AS 67%

End-to-End Active Speaker Detection

Juan Leon Alcazar, Moritz Cordes, Chen Zhao, Bernard Ghanem

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

2207.05692 2022-07-13 cs.MM cs.CL cs.SD eess.AS 67%

Lip-Listening: Mixing Senses to Understand Lips using Cross Modality Knowledge Distillation for Word-Based Models

Hadeel Mabrouk, Omar Abugabal, Nourhan Sakr, Hesham M. Eraqi

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments arXiv admin note: text overlap with arXiv:2108.03543

2206.07684 2022-06-16 cs.CV cs.MM cs.SD eess.AS 67%

AVATAR: Unconstrained Audiovisual Speech Recognition

Valentin Gabeur, Paul Hongsuck Seo, Arsha Nagrani, Chen Sun, Karteek Alahari, Cordelia Schmid

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

2203.02195 2022-06-14 cs.CV cs.AI cs.MM 67%

Voice-Face Homogeneity Tells Deepfake

Harry Cheng, Yangyang Guo, Tianyi Wang, Qi Li, Xiaojun Chang, Liqiang Nie

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 13 pages for peer review. Code will be released at https://github.com/xaCheng1996/VFD

2205.04274 2022-05-31 cs.CL cs.AI cs.CV 67%

Detecting and Understanding Harmful Memes: A Survey

Shivam Sharma, Firoj Alam, Md. Shad Akhtar, Dimitar Dimitrov, Giovanni Da San Martino, Hamed Firooz, Alon Halevy, Fabrizio Silvestri, Preslav Nakov, Tanmoy Chakraborty

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at IJCAI-ECAI 2022 (Survey Track) - Editorial Feedback Revised, 9 pages (7 main + 2 reference pages)

2201.02639 2022-05-16 cs.CV cs.CL cs.LG cs.SD eess.AS 67%

MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound

Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, Yejin Choi

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments CVPR 2022. Project page at https://rowanzellers.com/merlotreserve

2112.08995 2022-05-04 cs.SD cs.CL cs.CV eess.AS 67%

Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer

Yanpeng Zhao, Jack Hessel, Youngjae Yu, Ximing Lu, Rowan Zellers, Yejin Choi

专题命中音频语音多模态：image-text(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Accepted to NAACL 2022. Our code is available at https://github.com/zhaoyanpeng/vipant

2204.14272 2022-05-02 cs.CL cs.AI cs.SD eess.AS 67%

End-to-end Spoken Conversational Question Answering: Task, Dataset and Model

Chenyu You, Nuo Chen, Fenglin Liu, Shen Ge, Xian Wu, Yuexian Zou

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments In Findings of NAACL 2022. arXiv admin note: substantial text overlap with arXiv:2010.08923

2204.09919 2022-04-22 cs.HC cs.AI cs.MM cs.SD eess.AS 67%

Sonic Interactions in Virtual Environments: the Egocentric Audio Perspective of the Digital Twin

Michele Geronazzo, Stefania Serafin

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 46 pages, 5 figures. Pre-print version of the introduction to the book "Sonic Interactions in Virtual Environments" in press for Springer's Human-Computer Interaction Series, Open Access license. The pre-print editors' copy of the book can be found at https://vbn.aau.dk/en/publications/sonic-interactions-in-virtual-environments - full book info: https://sive.create.aau.dk/index.php/sivebook/

2204.01726 2022-04-06 cs.CV cs.AI eess.AS 67%

Lip to Speech Synthesis with Visual Context Attentional GAN

Minsu Kim, Joanna Hong, Yong Man Ro

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Published at NeurIPS 2021

2203.15991 2022-03-31 cs.CV cs.MM cs.SD eess.AS 67%

The Sound of Bounding-Boxes

Takashi Oya, Shohei Iwase, Shigeo Morishima

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 6 pages, 5 figures, ICPR (International Conference on Pattern Recognition) 2022

2203.09324 2022-03-30 cs.CV cs.LG cs.MM cs.SD eess.AS 67%

Localizing Visual Sounds the Easy Way

Shentong Mo, Pedro Morgado

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

1. 音频语音多模态 4597 篇

AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head

Hi Sheldon! Creating Deep Personalized Characters from TV Shows

Conditional Generation of Audio from Video via Foley Analogies

FlowGrad: Using Motion for Visual Sound Source Localization

The Music Annotation Pattern

SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision

Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samples

Learning Cross-lingual Visual Speech Representations

Towards Robust Image-in-Audio Deep Steganography

Multi-Modality in Music: Predicting Emotion in Music from High-Level Audio Features and Lyrics

Lip-to-Speech Synthesis in the Wild with Multi-task Learning

UAVM: Towards Unifying Audio and Visual Models

SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning

Symbol Emergence in Robotics: A Survey

LISA: Localized Image Stylization with Audio via Implicit Neural Representation

Automated Audio Captioning: An Overview of Recent Progress and New Challenges

The ReprGesture entry to the GENEA Challenge 2022

Speaker-adaptive Lip Reading with User-dependent Padding

End-to-End Active Speaker Detection

Lip-Listening: Mixing Senses to Understand Lips using Cross Modality Knowledge Distillation for Word-Based Models

AVATAR: Unconstrained Audiovisual Speech Recognition

Voice-Face Homogeneity Tells Deepfake

Detecting and Understanding Harmful Memes: A Survey

MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound

Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer

End-to-end Spoken Conversational Question Answering: Task, Dataset and Model

Sonic Interactions in Virtual Environments: the Egocentric Audio Perspective of the Digital Twin

Lip to Speech Synthesis with Visual Context Attentional GAN

The Sound of Bounding-Boxes

Localizing Visual Sounds the Easy Way