多模态大模型 - arXivDaily 专题

2110.08243 2022-03-16 eess.AS cs.CL cs.CV cs.LG cs.SD eess.IV 67%

Neural Dubber: Dubbing for Videos According to Scripts

Chenxu Hu, Qiao Tian, Tingle Li, Yuping Wang, Yuxuan Wang, Hang Zhao

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Accepted by NeurIPS 2021; Project page at https://tsinghua-mars-lab.github.io/NeuralDubber/

2202.10453 2022-02-23 cs.CV cs.LG cs.MM cs.SD eess.AS 67%

Predicting emotion from music videos: exploring the relative contribution of visual and auditory information to affective responses

Phoebe Chua, Dimos Makris, Dorien Herremans, Gemma Roig, Kat Agres

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 16 pages with 9 figures

2111.05846 2021-11-11 cs.SD cs.CV cs.MM cs.RO eess.AS 67%

Structure from Silence: Learning Scene Structure from Ambient Sound

Ziyang Chen, Xixi Hu, Andrew Owens

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to CoRL 2021 (Oral Presentation)

2105.08052 2021-10-26 cs.CV cs.MM cs.RO cs.SD eess.AS 67%

The Boombox: Visual Reconstruction from Acoustic Vibrations

Boyuan Chen, Mia Chiquier, Hod Lipson, Carl Vondrick

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments CoRL 2021. Website: boombox.cs.columbia.edu

2109.09105 2021-09-22 cs.CL cs.AI cs.LG eess.AS 67%

What BERT Based Language Models Learn in Spoken Transcripts: An Empirical Study

Ayush Kumar, Mukuntha Narayanan Sundararaman, Jithendra Vepa

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments BlackboxNLP @ EMNLP 2021 (15 pages, includes Appendix)

2108.02607 2021-08-06 cs.CV cs.MM cs.SD eess.AS eess.IV 67%

UniCon: Unified Context Network for Robust Active Speaker Detection

Yuanhang Zhang, Susan Liang, Shuang Yang, Xiao Liu, Zhongqin Wu, Shiguang Shan, Xilin Chen

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 10 pages, 6 figures; to appear at ACM Multimedia 2021

2107.09262 2021-07-21 cs.LG cs.AI cs.CV cs.MM cs.SD 67%

FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos

Sanchita Ghose, John J. Prevost

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments This article is under review in IEEE Transaction on Multimedia. It contains total 12 pages, 6 figures, 4 tables

2104.02687 2021-04-07 cs.CV cs.AI cs.MM 67%

Strumming to the Beat: Audio-Conditioned Contrastive Video Textures

Medhini Narasimhan, Shiry Ginosar, Andrew Owens, Alexei A. Efros, Trevor Darrell

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Project website at https://medhini.github.io/audio_video_textures/

2104.02026 2021-04-06 cs.CV cs.MM cs.SD eess.AS 67%

Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation

Yapeng Tian, Di Hu, Chenliang Xu

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments CVPR 2021

2005.08606 2021-03-22 cs.CV cs.MM cs.SD eess.AS 67%

End-to-End Lip Synchronisation Based on Pattern Classification

You Jin Kim, Hee Soo Heo, Soo-Whan Chung, Bong-Jin Lee

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments slt 2021 accepted

2005.01400 2021-03-19 eess.AS cs.CL cs.CV cs.LG 67%

Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition?

Abhinav Shukla, Stavros Petridis, Maja Pantic

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Accepted for publication in IEEE Transactions on Affective Computing; v3: Publication-ready version including additional experiments and discussion

2012.15635 2021-01-01 cs.MM cs.AI cs.CV 67%

Leveraging Audio Gestalt to Predict Media Memorability

Lorin Sweeney, Graham Healy, Alan F. Smeaton

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 3 pages, 1 Figure, 2 Tables

Journal ref MediaEval Multimedia Benchmark Workshop Working Notes, 14-15 December 2020

2003.08474 2020-12-22 eess.SP cs.CY cs.HC stat.AP 67%

TILES-2018, a longitudinal physiologic and behavioral data set of hospital workers

Karel Mundnich, Brandon M. Booth, Michelle L'Hommedieu, Tiantian Feng, Benjamin Girault, Justin L'Hommedieu, Mackenzie Wildman, Sophia Skaaden, Amrutha Nadarajan, Jennifer L. Villatte, Tiago H. Falk, Kristina Lerman, Emilio Ferrara, Shrikanth Narayanan

专题命中音频语音多模态：multimodal(abstract)；multi-modal(abstract)

Comments 57 pages, 9 figures, journal paper

Journal ref Sci Data 7, 354 (2020)

2009.05103 2020-09-14 cs.CV cs.MM cs.SD eess.AS 67%

Emotion-Based End-to-End Matching Between Image and Music in Valence-Arousal Space

Sicheng Zhao, Yaxian Li, Xingxu Yao, Weizhi Nie, Pengfei Xu, Jufeng Yang, Kurt Keutzer

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by ACM Multimedia 2020

2007.09902 2020-07-21 cs.CV cs.MM cs.SD eess.AS 67%

Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation

Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, Ziwei Liu

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments To appear in Proceedings of the European Conference on Computer Vision (ECCV), 2020. Code, models, and video results are available on our webpage: https://hangz-nju-cuhk.github.io/projects/Sep-Stereo

2004.09476 2020-04-21 cs.CV cs.LG cs.MM cs.SD eess.AS 67%

Music Gesture for Visual Sound Separation

Chuang Gan, Deng Huang, Hang Zhao, Joshua B. Tenenbaum, Antonio Torralba

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments CVPR 2020. Project page: http://music-gesture.csail.mit.edu

2003.00418 2020-03-03 cs.CV cs.AI cs.LG cs.MM cs.SD 67%

Towards Automatic Face-to-Face Translation

Prajwal K R, Rudrabha Mukhopadhyay, Jerin Philip, Abhishek Jha, Vinay Namboodiri, C. V. Jawahar

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 9 pages (including references), 5 figures, Published in ACM Multimedia, 2019

Journal ref MM '19: Proceedings of the 27th ACM International Conference on Multimedia; October 2019; Pages 1428-1436

2002.05639 2020-02-19 cs.CL cs.MM eess.AS 67%

Looking Enhances Listening: Recovering Missing Speech Using Images

Tejas Srinivasan, Ramon Sanabria, Florian Metze

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted to ICASSP 2020

1910.01990 2019-10-07 cs.CL cs.AI eess.AS 67%

Detecting Deception in Political Debates Using Acoustic and Textual Features

Daniel Kopev, Ahmed Ali, Ivan Koychev, Preslav Nakov

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Journal ref ASRU-2019

1907.01154 2019-07-03 cs.MM cs.AI cs.SD eess.AS 67%

Adaptive Music Composition for Games

Patrick Hutchings, Jon McCormack

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments Preprint. Accepted for publication in IEEE Transactions on Games, 2019

1809.02587 2018-09-10 cs.SD cs.CV cs.LG cs.MM eess.AS 67%

Self-Supervised Generation of Spatial Audio for 360 Video

Pedro Morgado, Nuno Vasconcelos, Timothy Langlois, Oliver Wang

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments To appear in NIPS 2018

1805.06652 2018-05-18 cs.HC 67%

Affective computing using speech and eye gaze: a review and bimodal system proposal for continuous affect prediction

Jonny O'Dwyer, Niall Murray, Ronan Flynn

专题命中音频语音多模态：multi-modal(abstract)；audio-visual(abstract)

Comments Submitted to International Journal of Human-Computer Studies

1710.01093 2018-04-26 cs.CV cs.CL eess.AS 67%

Which phoneme-to-viseme maps best improve visual-only computer lip-reading?

Helen L. Bear, Richard W. Harvey, Barry-John Theobald, Yuxuan Lan

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.CL、eess.AS

Journal ref Helen L. Bear, Richard W. Harvey, Barry-John Theobald, and Yuxuan Lan. Which phoneme-to-viseme maps best improve visual-only computer lip-reading? Advances in Visual Computing 2014. p230-239

1709.01683 2017-09-07 cs.HC 67%

Affect Recognition in Ads with Application to Computational Advertising

Abhinav Shukla, Shruti Shriya Gullapuram, Harish Katti, Karthik Yadati, Mohan Kankanhalli, Ramanathan Subramanian

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)

Comments Accepted at the ACM International Conference on Multimedia (ACM MM) 2017

1701.02477 2017-01-11 cs.CL cs.AI cs.CV cs.LG 67%

Multi-task Learning Of Deep Neural Networks For Audio Visual Automatic Speech Recognition

Abhinav Thanda, Shankar M Venkatesan

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.CL、cs.AI

1203.4422 2012-03-21 stat.ML cs.LG 67%

Semi-Supervised Single- and Multi-Domain Regression with Multi-Domain Training

Tomer Michaeli, Yonina C. Eldar, Guillermo Sapiro

专题命中音频语音多模态：multi-modal(abstract)；audio-visual(abstract)

Comments 24 pages, 6 figures, 2 tables

2409.13726 2024-09-24 cs.CL cs.AI cs.HC cs.LG 66%

Multilingual Dyadic Interaction Corpus NoXi+J: Toward Understanding Asian-European Non-verbal Cultural Characteristics and their Influences on Engagement

Marius Funk, Shogo Okada, Elisabeth André

专题命中音频语音多模态：multimodal(abstract,comments)；分类 cs.CL、cs.AI

Comments 8 pages. 6 figures. International Conference on Multimodal Interaction, November 4-8, 2024, San Jose, Costa Rica

2202.03007 2022-02-08 cs.CV cs.SD eess.AS eess.IV 66%

Learning Sound Localization Better From Semantically Similar Samples

Arda Senocak, Hyeonggon Ryu, Junsik Kim, In So Kweon

专题命中音频语音多模态：audio-visual(abstract,comments)；分类 cs.CV、eess.AS

Comments Accepted to ICASSP 2022. SOTA performance in Audio-Visual Sound Localization. 5 Pages

1806.05622 2020-11-05 cs.SD cs.CV eess.AS 66%

VoxCeleb2: Deep Speaker Recognition

Joon Son Chung, Arsha Nagrani, Andrew Zisserman

专题命中音频语音多模态：audio-visual(abstract,comments)；分类 cs.CV、eess.AS

Comments To appear in Interspeech 2018. The audio-visual dataset can be downloaded from http://www.robots.ox.ac.uk/~vgg/data/voxceleb2 . 1806.05622v2: minor fixes; 5 pages

2205.03802 2022-05-10 cs.CV 65%

Past and Future Motion Guided Network for Audio Visual Event Localization

Tingxiu Chen, Jianqin Yin, Jin Tang

专题命中音频语音多模态：audio-visual(abstract,comments)；分类 cs.CV；multi-modal(comments)

Comments event localization, multi-modal learning, audio-visual learning, audio attention, motion extraction

1. 音频语音多模态 4597 篇

Neural Dubber: Dubbing for Videos According to Scripts

Predicting emotion from music videos: exploring the relative contribution of visual and auditory information to affective responses

Structure from Silence: Learning Scene Structure from Ambient Sound

The Boombox: Visual Reconstruction from Acoustic Vibrations

What BERT Based Language Models Learn in Spoken Transcripts: An Empirical Study

UniCon: Unified Context Network for Robust Active Speaker Detection

FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos

Strumming to the Beat: Audio-Conditioned Contrastive Video Textures

Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation

End-to-End Lip Synchronisation Based on Pattern Classification

Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition?

Leveraging Audio Gestalt to Predict Media Memorability

TILES-2018, a longitudinal physiologic and behavioral data set of hospital workers

Emotion-Based End-to-End Matching Between Image and Music in Valence-Arousal Space

Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation

Music Gesture for Visual Sound Separation

Towards Automatic Face-to-Face Translation

Looking Enhances Listening: Recovering Missing Speech Using Images

Detecting Deception in Political Debates Using Acoustic and Textual Features

Adaptive Music Composition for Games

Self-Supervised Generation of Spatial Audio for 360 Video

Affective computing using speech and eye gaze: a review and bimodal system proposal for continuous affect prediction

Which phoneme-to-viseme maps best improve visual-only computer lip-reading?

Affect Recognition in Ads with Application to Computational Advertising

Multi-task Learning Of Deep Neural Networks For Audio Visual Automatic Speech Recognition

Semi-Supervised Single- and Multi-Domain Regression with Multi-Domain Training

Multilingual Dyadic Interaction Corpus NoXi+J: Toward Understanding Asian-European Non-verbal Cultural Characteristics and their Influences on Engagement

Learning Sound Localization Better From Semantically Similar Samples

VoxCeleb2: Deep Speaker Recognition

Past and Future Motion Guided Network for Audio Visual Event Localization