多模态大模型

Evaluation of pretrained language models on music understanding

Yannis Vasilakis, Rachel Bittner, Johan Pauwels

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、eess.AS

Efficient Video to Audio Mapper with Visual Scene Detection

Mingjing Yi, Ming Li

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.MM、eess.AS

DSCLAP: Domain-Specific Contrastive Language-Audio Pre-Training

Shengqiang Liu, Da Liu, Anna Wang, Zhiyu Zhang, Jie Gao, Yali Li

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.MM、eess.AS

ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds

Sreyan Ghosh, Sonal Kumar, Chandra Kiran Reddy Evuru, Oriol Nieto, Ramani Duraiswami, Dinesh Manocha

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments Code and Checkpoints: https://github.com/Sreyan88/ReCLAP

LLaQo: Towards a Query-Based Coach in Expressive Music Performance Assessment

Huan Zhang, Vincent Cheung, Hayato Nishioka, Simon Dixon, Shinichi Furuya

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.MM、eess.AS

Affective Computing Has Changed: The Foundation Model Disruption

Björn Schuller, Adria Mallol-Ragolta, Alejandro Peña Almansa, Iosif Tsangko, Mostafa M. Amin, Anastasia Semertzidou, Lukas Christ, Shahin Amiriparian

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

VoiceWukong: Benchmarking Deepfake Voice Detection

Ziwei Yan, Yanjie Zhao, Haoyu Wang

专题命中 音频语音多模态 ：MLLM(abstract)；分类 cs.AI、eess.AS

Applications and Advances of Artificial Intelligence in Music Generation:A Review

Yanxu Chen, Linshu Huang, Tian Gou

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、eess.AS

FastVoiceGrad: One-step Diffusion-Based Voice Conversion with Adversarial Conditional Diffusion Distillation

Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, Yuto Kondo

专题命中 音频语音多模态 ：any-to-any(abstract)；分类 cs.AI、eess.AS

Comments Accepted to Interspeech 2024. Project page: https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/fastvoicegrad/

Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent

Shanbo Cheng, Zhichao Huang, Tom Ko, Hang Li, Ningxin Peng, Lu Xu, Qini Zhang

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments Authors are listed in alphabetical order by last name. Demonstrations and human-annotated test sets are available at https://byteresearchcla.github.io/clasi

ECC Analyzer: Extract Trading Signal from Earnings Conference Calls using Large Language Model for Stock Performance Prediction

Yupeng Cao, Zhi Chen, Qingyun Pei, Nathan Jinseok Lee, K. P. Subbalakshmi, Papa Momar Ndiaye

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 9 pages, 1 figures, 2 tables

SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Description

Zeyu Jin, Jia Jia, Qixin Wang, Kehan Li, Shuoyi Zhou, Songtao Zhou, Xiaoyu Qin, Zhiyong Wu

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、cs.MM

Comments Accepted by ACM Multimedia 2024

Speech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyond

Beomseok Lee, Ioan Calapodescu, Marco Gaido, Matteo Negri, Laurent Besacier

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted at INTERSPEECH 2024. This version includes the same content but with additional appendices

ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performer

Jiazhi Guan, Zhiliang Xu, Hang Zhou, Kaisiyuan Wang, Shengyi He, Zhanwang Zhang, Borong Liang, Haocheng Feng, Errui Ding, Jingtuo Liu, Jingdong Wang, Youjian Zhao, Ziwei Liu

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM

Comments Accepted to European Conference on Computer Vision (ECCV), 2024. Project page: https://guanjz20.github.io/projects/ReSyncer

EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving

Jiacheng Lin, Jiajun Chen, Kunyu Peng, Xuan He, Zhiyong Li, Rainer Stiefelhagen, Kailun Yang

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted to IEEE Transactions on Intelligent Transportation Systems (T-ITS). The source code and datasets are available at https://github.com/lab206/EchoTrack

AMA-LSTM: Pioneering Robust and Fair Financial Audio Analysis for Stock Volatility Prediction

Shengkun Wang, Taoran Ji, Jianfeng He, Mariam Almutairi, Dan Wang, Linhan Wang, Min Zhang, Chang-Tien Lu

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Modeling and Driving Human Body Soundfields through Acoustic Primitives

Chao Huang, Dejan Markovic, Chenliang Xu, Alexander Richard

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments ECCV 2024. Project Page: https://wikichao.github.io/Acoustic-Primitives/

Self-supervised ASR Models and Features For Dysarthric and Elderly Speech Recognition

Shujie Hu, Xurong Xie, Mengzhe Geng, Zengrui Jin, Jiajun Deng, Guinan Li, Yi Wang, Mingyu Cui, Tianzi Wang, Helen Meng, Xunying Liu

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.AI、eess.AS

Comments IEEE/ACM Transactions on Audio, Speech, and Language Processing

Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representation

Minsu Kim, Jeong Hun Yeo, Se Jin Park, Hyeongseop Rha, Yong Man Ro

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments ACMMM 2024

Scaling Sign Language Translation

Biao Zhang, Garrett Tanzer, Orhan Firat

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL

Qwen2-Audio Technical Report

Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, Jingren Zhou

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments https://github.com/QwenLM/Qwen2-Audio. Checkpoints, codes and scripts will be opensoursed soon

Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in Tunisian Dialect

Salima Mdhaffar, Haroun Elleuch, Fethi Bougares, Yannick Estève

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted in ArabicNLP 2024

Music Era Recognition Using Supervised Contrastive Learning and Artist Information

Qiqi He, Xuchen Song, Weituo Hao, Ju-Chiang Wang, Wei-Tsung Lu, Wei Li

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、eess.AS

T-VSL: Text-Guided Visual Sound Source Localization in Mixtures

Tanvir Mahmud, Yapeng Tian, Diana Marculescu

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted in CVPR-2024. Code: https://github.com/enyac-group/T-VSL/tree/main

Journal ref IEEE/CVF Computer Vision and Pattern Recognition (CVPR) Conference, 2024

The Solution for Temporal Sound Localisation Task of ICCV 1st Perception Test Challenge 2023

Yurui Huang, Yang Yang, Shou Chen, Xiangyu Wu, Qingguo Chen, Jianfeng Lu

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、eess.AS

FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds

Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, Kai Chen

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Project page: https://foleycrafter.github.io/

This Paper Had the Smartest Reviewers -- Flattery Detection Utilising an Audio-Textual Transformer-Based Approach

Lukas Christ, Shahin Amiriparian, Friederike Hawighorst, Ann-Kathrin Schill, Angelo Boutalikakis, Lorenz Graf-Vlachy, Andreas König, Björn W. Schuller

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Interspeech 2024

CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving

Bhavani Shankar, Preethi Jyothi, Pushpak Bhattacharyya

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

BTS: Bridging Text and Sound Modalities for Metadata-Aided Respiratory Sound Classification

June-Woo Kim, Miika Toikkanen, Yera Choi, Seoung-Eun Moon, Ho-Young Jung

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、eess.AS

Comments Accepted INTERSPEECH 2024

PolySpeech: Exploring Unified Multitask Speech Models for Competitiveness with Single-task Models

Runyan Yang, Huibao Yang, Xiqing Zhang, Tiantian Ye, Ying Liu, Yingying Gao, Shilei Zhang, Chao Deng, Junlan Feng

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments 5 pages, 2 figures