多模态大模型

Speech inpainting: Context-based speech synthesis guided by video

Juan F. Montesinos, Daniel Michelsanti, Gloria Haro, Zheng-Hua Tan, Jesper Jensen

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.AI、eess.AS

Comments Accepted in Interspeech23

Inspecting Spoken Language Understanding from Kids for Basic Math Learning at Home

Eda Okur, Roddy Fuentes Alba, Saurav Sahay, Lama Nachman

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA) at ACL 2023

Voice Conversion With Just Nearest Neighbors

Matthew Baas, Benjamin van Niekerk, Herman Kamper

专题命中 音频语音多模态 ：any-to-any(abstract)；分类 cs.CL、eess.AS

Comments 5 page, 1 table, 2 figures. Accepted at Interspeech 2023

VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation

Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, Furu Wei

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments Working in progress

Exploring Train and Test-Time Augmentations for Audio-Language Learning

Eungbeom Kim, Jinhee Kim, Yoori Oh, Kyungsu Kim, Minju Park, Jaeheon Sim, Jinwoo Lee, Kyogu Lee

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments 5 pages, 4 figures

DiffAVA: Personalized Text-to-Audio Generation with Visual Alignment

Shentong Mo, Jing Shi, Yapeng Tian

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.MM

Points2Sound: From mono to binaural audio using 3D point cloud scenes

Francesc Lluís, Vasileios Chatziioannou, Alex Hofmann

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、eess.AS

Comments Code, data, and listening examples: https://github.com/francesclluis/points2sound

Journal ref EURASIP Journal on Audio, Speech, and Music Processing 2022 (1), 1-15

Understanding and Bridging the Modality Gap for Speech Translation

Qingkai Fang, Yang Feng

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments ACL 2023 main conference

Emotional Expression Detection in Spoken Language Employing Machine Learning Algorithms

Mehrab Hosain, Most. Yeasmin Arafat, Gazi Zahirul Islam, Jia Uddin, Md. Mobarak Hossain, Fatema Alam

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Journal Pre-print (15 Pages, 9 Figures, 3 Tables)

Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations

Sagnik Majumder, Hao Jiang, Pierre Moulon, Ethan Henderson, Paul Calamia, Kristen Grauman, Vamsi Krishna Ithapu

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted to CVPR 2023

An Empirical Study and Improvement for Speech Emotion Recognition

Zhen Wu, Yizhe Lu, Xinyu Dai

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted by ICASSP 2023

Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert

Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T. Tan, Haizhou Li

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.AI

Comments accepted by CVPR 2023

AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR

Paul Hongsuck Seo, Arsha Nagrani, Cordelia Schmid

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、eess.AS

Comments CVPR 2023

Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation

Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, Lequan Yu

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CV、eess.AS

Comments Accepted by IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023. 10 pages, 3 figures

FaceChat: An Emotion-Aware Face-to-face Dialogue Framework

Deema Alnuhait, Qingyang Wu, Zhou Yu

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

wav2vec and its current potential to Automatic Speech Recognition in German for the usage in Digital History: A comparative assessment of available ASR-technologies for the use in cultural heritage contexts

Michael Fleck, Wolfgang Göderle

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CL、eess.AS

Comments 11 pages, 2 tables

A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT

Yihan Cao, Siyu Li, Yixin Liu, Zhiling Yan, Yutong Dai, Philip S. Yu, Lichao Sun

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 44 pages, 15 figures

Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysis

Vikramjit Mitra, Vasudha Kowtha, Hsiang-Yun Sherry Chien, Erdrin Azemi, Carlos Avendano

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments 5 pages, conference

TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation

Rongjie Huang, Jinglin Liu, Huadai Liu, Yi Ren, Lichao Zhang, Jinzheng He, Zhou Zhao

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accpeted to ICLR 2023

Imaginary Voice: Face-styled Diffusion Model for Text-to-Speech

Jiyoung Lee, Joon Son Chung, Soo-Whan Chung

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments ICASSP 2023. Project page: https://facetts.github.io

SSVMR: Saliency-based Self-training for Video-Music Retrieval

Xuxin Cheng, Zhihong Zhu, Hongxiang Li, Yaowei Li, Yuexian Zou

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.MM、eess.AS

Comments Accepted by ICASSP 2023

Sound Localization by Self-Supervised Time Delay Estimation

Ziyang Chen, David F. Fouhey, Andrew Owens

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、eess.AS

Comments ECCV 2022

Detecting Dementia from Speech and Transcripts using Transformers

Loukas Ilias, Dimitris Askounis, John Psarras

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments Computer Speech & Language (Accepted)

Journal ref Computer Speech & Language (Volume: 79, April 2023)

Exploring the Efficacy of Pre-trained Checkpoints in Text-to-Music Generation Task

Shangda Wu, Maosong Sun

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted by the Creative AI Across Modalities workshop at AAAI 2023

Audiovisual Database with 360 Video and Higher-Order Ambisonics Audio for Perception, Cognition, Behavior, and QoE Evaluation Research

Thomas Robotham, Ashutosh Singla, Olli S. Rummukainen, Alexander Raake, Emanuël A. P. Habets

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.MM、eess.AS

Comments 6 pages, 2 figures, accepted and presented at the 2022 14th International Conference on Quality of Multimedia Experience (QoMEX). Database is publicly accessible at https://qoevave.github.io/database/

ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement

Wei-Ning Hsu, Tal Remez, Bowen Shi, Jacob Donley, Yossi Adi

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Toward Universal Text-to-Music Retrieval

SeungHeon Doh, Minz Won, Keunwoo Choi, Juhan Nam

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.MM、eess.AS

The applicability of transperceptual and deep learning approaches to the study and mimicry of complex cartilaginous tissues

J. Waghorne, C. Howard, H. Hu, J. Pang, W. J. Peveler, L. Harris, O. Barrera

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Proceedings of the 4th International Workshop on Reading Music Systems

Jorge Calvo-Zaragoza, Alexander Pacha, Elona Shatri

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、eess.AS

Comments Proceedings edited by Jorge Calvo-Zaragoza, Alexander Pacha and Elona Shatri

Self-Supervised Speech Representation Learning: A Review

Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D. Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, Tara N. Sainath, Shinji Watanabe

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS