多模态大模型

My Science Tutor (MyST) -- A Large Corpus of Children's Conversational Speech

Sameer S. Pradhan, Ronald A. Cole, Wayne H. Ward

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

FoleyGen: Visually-Guided Audio Generation

Xinhao Mei, Varun Nagaraja, Gael Le Lan, Zhaoheng Ni, Ernie Chang, Yangyang Shi, Vikas Chandra

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.MM、eess.AS

Addressing the Blind Spots in Spoken Language Processing

Amit Moryossef

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments 5 pages

UniBriVL: Robust Universal Representation and Generation of Audio Driven Diffusion Models

Sen Fang, Bowen Gao, Yangjian Wu, Teik Toe Teoh

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、eess.AS

Comments Voice-Text fusion input; The first work of audio driven diffusion model. arXiv admin note: text overlap with arXiv:2303.04585

Comparative Analysis of Deep-Fake Algorithms

Nikhil Sontakke, Sejal Utekar, Shivansh Rastogi, Shriraj Sonawane

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.AI

Comments 7 pages, 4 figures, 2 tables, Published with International Journal of Computer Science Trends and Technology (IJCST)

Journal ref International Journal of Computer Science Trends and Technology (IJCST) V11(4): Page(109-115) Jul - Aug 2023. ISSN: 2347-8578

A Comprehensive Survey for Evaluation Methodologies of AI-Generated Music

Zeyu Xiong, Weitao Wang, Jing Yu, Yue Lin, Ziyan Wang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、eess.AS

Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation

Ziyang Chen, Shengyi Qian, Andrew Owens

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments ICCV 2023. Project site: https://ificl.github.io/SLfM/

Radio2Text: Streaming Speech Recognition Using mmWave Radio Signals

Running Zhao, Jiangtao Yu, Hang Zhao, Edith C. H. Ngai

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments Accepted by Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (ACM IMWUT/UbiComp 2023)

Whose Emotion Matters? Speaking Activity Localisation without Prior Knowledge

Hugo Carneiro, Cornelius Weber, Stefan Wermter

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、eess.AS

Comments 17 pages, 8 figures, 7 tables, Published in Neurocomputing

Journal ref Neurocomputing (2023); Volume 545; 126271

Capturing Spectral and Long-term Contextual Information for Speech Emotion Recognition Using Deep Learning Techniques

Samiul Islam, Md. Maksudul Haque, Abu Jobayer Md. Sadat

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments the research paper is still in progress

Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis

Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, Chen Zhang, Xiang Yin, Zejun Ma, Zhou Zhao

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted by ICML 2023 Workshop, 6 pages, 3 figures

The Hidden Dance of Phonemes and Visage: Unveiling the Enigmatic Link between Phonemes and Facial Features

Liao Qu, Xianwei Zou, Xiang Li, Yandong Wen, Rita Singh, Bhiksha Raj

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、eess.AS

Comments Interspeech 2023

CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Models

Hao-Wen Dong, Xiaoyu Liu, Jordi Pons, Gautam Bhattacharya, Santiago Pascual, Joan Serrà, Taylor Berg-Kirkpatrick, Julian McAuley

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.MM、eess.AS

Comments Accepted by WASPAA 2023. Demo: https://salu133445.github.io/clipsonic/

Topic Identification For Spontaneous Speech: Enriching Audio Features With Embedded Linguistic Information

Dejan Porjazovski, Tamás Grósz, Mikko Kurimo

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments Accepted to EUSIPCO 2023

Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models

Simian Luo, Chuanhao Yan, Chenxu Hu, Hang Zhao

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Classification of Infant Sleep/Wake States: Cross-Attention among Large Scale Pretrained Transformer Networks using Audio, ECG, and IMU Data

Kai Chieh Chang, Mark Hasegawa-Johnson, Nancy L. McElwain, Bashima Islam

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.MM、eess.AS

Comments Preprint for APSIPA2023

Mu$^{2}$SLAM: Multitask, Multilingual Speech and Language Models

Yong Cheng, Yu Zhang, Melvin Johnson, Wolfgang Macherey, Ankur Bapna

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments ICML 2023

Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognition

Shujie Hu, Xurong Xie, Zengrui Jin, Mengzhe Geng, Yi Wang, Mingyu Cui, Jiajun Deng, Xunying Liu, Helen Meng

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.AI、eess.AS

Comments accepted by ICASSP 2023

PATCorrect: Non-autoregressive Phoneme-augmented Transformer for ASR Error Correction

Ziji Zhang, Zhehui Wang, Rajesh Kamma, Sharanya Eswaran, Narayanan Sadagopan

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments Accepted camera-ready version for INTERSPEECH 2023

Visual-Aware Text-to-Speech

Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, Tiejun Zhao, Tao Mei

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments accepted as oral and top 3% paper by ICASSP 2023

Journal ref ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2023, 1-5

RealImpact: A Dataset of Impact Sound Fields for Real Objects

Samuel Clarke, Ruohan Gao, Mason Wang, Mark Rau, Julia Xu, Jui-Hsien Wang, Doug L. James, Jiajun Wu

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments CVPR 2023 (Highlight). Project page: https://samuelpclarke.com/realimpact/

OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment

Xize Cheng, Tao Jin, Linjun Li, Wang Lin, Xinyu Duan, Zhou Zhao

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.CL

Comments Accepted to ACL2023 (Oral)

Speech inpainting: Context-based speech synthesis guided by video

Juan F. Montesinos, Daniel Michelsanti, Gloria Haro, Zheng-Hua Tan, Jesper Jensen

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.AI、eess.AS

Comments Accepted in Interspeech23

Inspecting Spoken Language Understanding from Kids for Basic Math Learning at Home

Eda Okur, Roddy Fuentes Alba, Saurav Sahay, Lama Nachman

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA) at ACL 2023

Voice Conversion With Just Nearest Neighbors

Matthew Baas, Benjamin van Niekerk, Herman Kamper

专题命中 音频语音多模态 ：any-to-any(abstract)；分类 cs.CL、eess.AS

Comments 5 page, 1 table, 2 figures. Accepted at Interspeech 2023

VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation

Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, Furu Wei

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments Working in progress

Exploring Train and Test-Time Augmentations for Audio-Language Learning

Eungbeom Kim, Jinhee Kim, Yoori Oh, Kyungsu Kim, Minju Park, Jaeheon Sim, Jinwoo Lee, Kyogu Lee

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments 5 pages, 4 figures

DiffAVA: Personalized Text-to-Audio Generation with Visual Alignment

Shentong Mo, Jing Shi, Yapeng Tian

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.MM

Points2Sound: From mono to binaural audio using 3D point cloud scenes

Francesc Lluís, Vasileios Chatziioannou, Alex Hofmann

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、eess.AS

Comments Code, data, and listening examples: https://github.com/francesclluis/points2sound

Journal ref EURASIP Journal on Audio, Speech, and Music Processing 2022 (1), 1-15

Understanding and Bridging the Modality Gap for Speech Translation

Qingkai Fang, Yang Feng

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments ACL 2023 main conference