多模态大模型

FastVoiceGrad: One-step Diffusion-Based Voice Conversion with Adversarial Conditional Diffusion Distillation

Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, Yuto Kondo

专题命中 音频语音多模态 ：any-to-any(abstract)；分类 cs.AI、eess.AS

Comments Accepted to Interspeech 2024. Project page: https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/fastvoicegrad/

Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent

Shanbo Cheng, Zhichao Huang, Tom Ko, Hang Li, Ningxin Peng, Lu Xu, Qini Zhang

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments Authors are listed in alphabetical order by last name. Demonstrations and human-annotated test sets are available at https://byteresearchcla.github.io/clasi

ECC Analyzer: Extract Trading Signal from Earnings Conference Calls using Large Language Model for Stock Performance Prediction

Yupeng Cao, Zhi Chen, Qingyun Pei, Nathan Jinseok Lee, K. P. Subbalakshmi, Papa Momar Ndiaye

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 9 pages, 1 figures, 2 tables

SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Description

Zeyu Jin, Jia Jia, Qixin Wang, Kehan Li, Shuoyi Zhou, Songtao Zhou, Xiaoyu Qin, Zhiyong Wu

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、cs.MM

Comments Accepted by ACM Multimedia 2024

Speech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyond

Beomseok Lee, Ioan Calapodescu, Marco Gaido, Matteo Negri, Laurent Besacier

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted at INTERSPEECH 2024. This version includes the same content but with additional appendices

ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performer

Jiazhi Guan, Zhiliang Xu, Hang Zhou, Kaisiyuan Wang, Shengyi He, Zhanwang Zhang, Borong Liang, Haocheng Feng, Errui Ding, Jingtuo Liu, Jingdong Wang, Youjian Zhao, Ziwei Liu

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM

Comments Accepted to European Conference on Computer Vision (ECCV), 2024. Project page: https://guanjz20.github.io/projects/ReSyncer

EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving

Jiacheng Lin, Jiajun Chen, Kunyu Peng, Xuan He, Zhiyong Li, Rainer Stiefelhagen, Kailun Yang

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted to IEEE Transactions on Intelligent Transportation Systems (T-ITS). The source code and datasets are available at https://github.com/lab206/EchoTrack

AMA-LSTM: Pioneering Robust and Fair Financial Audio Analysis for Stock Volatility Prediction

Shengkun Wang, Taoran Ji, Jianfeng He, Mariam Almutairi, Dan Wang, Linhan Wang, Min Zhang, Chang-Tien Lu

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Modeling and Driving Human Body Soundfields through Acoustic Primitives

Chao Huang, Dejan Markovic, Chenliang Xu, Alexander Richard

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments ECCV 2024. Project Page: https://wikichao.github.io/Acoustic-Primitives/

Self-supervised ASR Models and Features For Dysarthric and Elderly Speech Recognition

Shujie Hu, Xurong Xie, Mengzhe Geng, Zengrui Jin, Jiajun Deng, Guinan Li, Yi Wang, Mingyu Cui, Tianzi Wang, Helen Meng, Xunying Liu

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.AI、eess.AS

Comments IEEE/ACM Transactions on Audio, Speech, and Language Processing

Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representation

Minsu Kim, Jeong Hun Yeo, Se Jin Park, Hyeongseop Rha, Yong Man Ro

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments ACMMM 2024

Scaling Sign Language Translation

Biao Zhang, Garrett Tanzer, Orhan Firat

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL

Qwen2-Audio Technical Report

Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, Jingren Zhou

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments https://github.com/QwenLM/Qwen2-Audio. Checkpoints, codes and scripts will be opensoursed soon

Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in Tunisian Dialect

Salima Mdhaffar, Haroun Elleuch, Fethi Bougares, Yannick Estève

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted in ArabicNLP 2024

Music Era Recognition Using Supervised Contrastive Learning and Artist Information

Qiqi He, Xuchen Song, Weituo Hao, Ju-Chiang Wang, Wei-Tsung Lu, Wei Li

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、eess.AS

T-VSL: Text-Guided Visual Sound Source Localization in Mixtures

Tanvir Mahmud, Yapeng Tian, Diana Marculescu

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted in CVPR-2024. Code: https://github.com/enyac-group/T-VSL/tree/main

Journal ref IEEE/CVF Computer Vision and Pattern Recognition (CVPR) Conference, 2024

The Solution for Temporal Sound Localisation Task of ICCV 1st Perception Test Challenge 2023

Yurui Huang, Yang Yang, Shou Chen, Xiangyu Wu, Qingguo Chen, Jianfeng Lu

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、eess.AS

FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds

Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, Kai Chen

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Project page: https://foleycrafter.github.io/

This Paper Had the Smartest Reviewers -- Flattery Detection Utilising an Audio-Textual Transformer-Based Approach

Lukas Christ, Shahin Amiriparian, Friederike Hawighorst, Ann-Kathrin Schill, Angelo Boutalikakis, Lorenz Graf-Vlachy, Andreas König, Björn W. Schuller

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Interspeech 2024

CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving

Bhavani Shankar, Preethi Jyothi, Pushpak Bhattacharyya

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

BTS: Bridging Text and Sound Modalities for Metadata-Aided Respiratory Sound Classification

June-Woo Kim, Miika Toikkanen, Yera Choi, Seoung-Eun Moon, Ho-Young Jung

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、eess.AS

Comments Accepted INTERSPEECH 2024

PolySpeech: Exploring Unified Multitask Speech Models for Competitiveness with Single-task Models

Runyan Yang, Huibao Yang, Xiqing Zhang, Tiantian Ye, Ying Liu, Yingying Gao, Shilei Zhang, Chao Deng, Junlan Feng

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments 5 pages, 2 figures

BLSP-Emo: Towards Empathetic Large Speech-Language Models

Chen Wang, Minpeng Liao, Zhongqiang Huang, Junhong Wu, Chengqing Zong, Jiajun Zhang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Reacting like Humans: Incorporating Intrinsic Human Behaviors into NAO through Sound-Based Reactions to Fearful and Shocking Events for Enhanced Sociability

Ali Ghadami, Mohammadreza Taghimohammadi, Mohammad Mohammadzadeh, Mohammad Hosseinipour, Alireza Taheri

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.AI、eess.AS

Comments 16 pages, 11 figures

Continual Contrastive Spoken Language Understanding

Umberto Cappellazzo, Enrico Fini, Muqiao Yang, Daniele Falavigna, Alessio Brutti, Bhiksha Raj

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、eess.AS

Comments Accepted to ACL Findings 2024

Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM

Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, Michelle Tadmor Ramanovich

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments ICLR 2024 camera-ready

NeRF-AD: Neural Radiance Field with Attention-based Disentanglement for Talking Face Synthesis

Chongke Bi, Xiaoxing Liu, Zhilei Liu

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.MM

Comments Accepted by ICASSP 2024

SwapTalk: Audio-Driven Talking Face Generation with One-Shot Customization in Latent Space

Zeren Zhang, Haibo Qin, Jiayu Huang, Yixin Li, Hui Lin, Yitao Duan, Jinwen Ma

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.AI

Large Language Models for Time Series: A Survey

Xiyuan Zhang, Ranak Roy Chowdhury, Rajesh K. Gupta, Jingbo Shang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments GitHub repository: https://github.com/xiyuanzh/awesome-llm-time-series

Visually grounded few-shot word learning in low-resource settings

Leanne Nortje, Dan Oneata, Herman Kamper

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted to TASLP. arXiv admin note: substantial text overlap with arXiv:2305.15937