多模态大模型

Emotional Expression Detection in Spoken Language Employing Machine Learning Algorithms

Mehrab Hosain, Most. Yeasmin Arafat, Gazi Zahirul Islam, Jia Uddin, Md. Mobarak Hossain, Fatema Alam

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Journal Pre-print (15 Pages, 9 Figures, 3 Tables)

Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations

Sagnik Majumder, Hao Jiang, Pierre Moulon, Ethan Henderson, Paul Calamia, Kristen Grauman, Vamsi Krishna Ithapu

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted to CVPR 2023

An Empirical Study and Improvement for Speech Emotion Recognition

Zhen Wu, Yizhe Lu, Xinyu Dai

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted by ICASSP 2023

Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert

Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T. Tan, Haizhou Li

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.AI

Comments accepted by CVPR 2023

AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR

Paul Hongsuck Seo, Arsha Nagrani, Cordelia Schmid

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、eess.AS

Comments CVPR 2023

Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation

Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, Lequan Yu

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CV、eess.AS

Comments Accepted by IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023. 10 pages, 3 figures

FaceChat: An Emotion-Aware Face-to-face Dialogue Framework

Deema Alnuhait, Qingyang Wu, Zhou Yu

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

wav2vec and its current potential to Automatic Speech Recognition in German for the usage in Digital History: A comparative assessment of available ASR-technologies for the use in cultural heritage contexts

Michael Fleck, Wolfgang Göderle

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CL、eess.AS

Comments 11 pages, 2 tables

A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT

Yihan Cao, Siyu Li, Yixin Liu, Zhiling Yan, Yutong Dai, Philip S. Yu, Lichao Sun

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 44 pages, 15 figures

Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysis

Vikramjit Mitra, Vasudha Kowtha, Hsiang-Yun Sherry Chien, Erdrin Azemi, Carlos Avendano

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments 5 pages, conference

TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation

Rongjie Huang, Jinglin Liu, Huadai Liu, Yi Ren, Lichao Zhang, Jinzheng He, Zhou Zhao

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accpeted to ICLR 2023

Imaginary Voice: Face-styled Diffusion Model for Text-to-Speech

Jiyoung Lee, Joon Son Chung, Soo-Whan Chung

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments ICASSP 2023. Project page: https://facetts.github.io

SSVMR: Saliency-based Self-training for Video-Music Retrieval

Xuxin Cheng, Zhihong Zhu, Hongxiang Li, Yaowei Li, Yuexian Zou

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.MM、eess.AS

Comments Accepted by ICASSP 2023

Sound Localization by Self-Supervised Time Delay Estimation

Ziyang Chen, David F. Fouhey, Andrew Owens

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、eess.AS

Comments ECCV 2022

Detecting Dementia from Speech and Transcripts using Transformers

Loukas Ilias, Dimitris Askounis, John Psarras

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments Computer Speech & Language (Accepted)

Journal ref Computer Speech & Language (Volume: 79, April 2023)

Exploring the Efficacy of Pre-trained Checkpoints in Text-to-Music Generation Task

Shangda Wu, Maosong Sun

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted by the Creative AI Across Modalities workshop at AAAI 2023

Audiovisual Database with 360 Video and Higher-Order Ambisonics Audio for Perception, Cognition, Behavior, and QoE Evaluation Research

Thomas Robotham, Ashutosh Singla, Olli S. Rummukainen, Alexander Raake, Emanuël A. P. Habets

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.MM、eess.AS

Comments 6 pages, 2 figures, accepted and presented at the 2022 14th International Conference on Quality of Multimedia Experience (QoMEX). Database is publicly accessible at https://qoevave.github.io/database/

ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement

Wei-Ning Hsu, Tal Remez, Bowen Shi, Jacob Donley, Yossi Adi

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Toward Universal Text-to-Music Retrieval

SeungHeon Doh, Minz Won, Keunwoo Choi, Juhan Nam

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.MM、eess.AS

The applicability of transperceptual and deep learning approaches to the study and mimicry of complex cartilaginous tissues

J. Waghorne, C. Howard, H. Hu, J. Pang, W. J. Peveler, L. Harris, O. Barrera

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Proceedings of the 4th International Workshop on Reading Music Systems

Jorge Calvo-Zaragoza, Alexander Pacha, Elona Shatri

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、eess.AS

Comments Proceedings edited by Jorge Calvo-Zaragoza, Alexander Pacha and Elona Shatri

Self-Supervised Speech Representation Learning: A Review

Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D. Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, Tara N. Sainath, Shinji Watanabe

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory

Se Jin Park, Minsu Kim, Joanna Hong, Jeongsoo Choi, Yong Man Ro

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.AI

Comments Accepted at AAAI 2022 (Oral)

No-audio speaking status detection in crowded settings via visual pose-based filtering and wearable acceleration

Jose Vargas-Quiros, Laura Cabrera-Quiros, Hayley Hung

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、eess.AS

Visual Speech Recognition for Multiple Languages in the Wild

Pingchuan Ma, Stavros Petridis, Maja Pantic

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Published in Nature Machine Intelligence

On the Role of Visual Context in Enriching Music Representations

Kleanthis Avramidis, Shanti Stewart, Shrikanth Narayanan

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.MM、eess.AS

Comments 5 pages, 4 figures, 1 table

Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddings

Jian Zhu, Zuoyu Tian, Yadong Liu, Cong Zhang, Chia-wen Lo

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Findings of EMNLP 2022

Foundation Transformers

Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, Barun Patra, Zhun Liu, Vishrav Chaudhary, Xia Song, Furu Wei

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments Work in progress

Match Cutting: Finding Cuts with Smooth Visual Transitions

Boris Chen, Amir Ziai, Rebecca Tucker, Yuchen Xie

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM

SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training

Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, Furu Wei

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments 14 pages, accepted by EMNLP 2022