多模态大模型

BLSP-Emo: Towards Empathetic Large Speech-Language Models

Chen Wang, Minpeng Liao, Zhongqiang Huang, Junhong Wu, Chengqing Zong, Jiajun Zhang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Reacting like Humans: Incorporating Intrinsic Human Behaviors into NAO through Sound-Based Reactions to Fearful and Shocking Events for Enhanced Sociability

Ali Ghadami, Mohammadreza Taghimohammadi, Mohammad Mohammadzadeh, Mohammad Hosseinipour, Alireza Taheri

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.AI、eess.AS

Comments 16 pages, 11 figures

Continual Contrastive Spoken Language Understanding

Umberto Cappellazzo, Enrico Fini, Muqiao Yang, Daniele Falavigna, Alessio Brutti, Bhiksha Raj

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、eess.AS

Comments Accepted to ACL Findings 2024

Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM

Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, Michelle Tadmor Ramanovich

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments ICLR 2024 camera-ready

NeRF-AD: Neural Radiance Field with Attention-based Disentanglement for Talking Face Synthesis

Chongke Bi, Xiaoxing Liu, Zhilei Liu

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.MM

Comments Accepted by ICASSP 2024

SwapTalk: Audio-Driven Talking Face Generation with One-Shot Customization in Latent Space

Zeren Zhang, Haibo Qin, Jiayu Huang, Yixin Li, Hui Lin, Yitao Duan, Jinwen Ma

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.AI

Large Language Models for Time Series: A Survey

Xiyuan Zhang, Ranak Roy Chowdhury, Rajesh K. Gupta, Jingbo Shang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments GitHub repository: https://github.com/xiyuanzh/awesome-llm-time-series

Visually grounded few-shot word learning in low-resource settings

Leanne Nortje, Dan Oneata, Herman Kamper

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted to TASLP. arXiv admin note: substantial text overlap with arXiv:2305.15937

AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs

Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, Mike Seltzer

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、cs.AI

GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing

Hao Lu, Xuesong Niu, Jiyao Wang, Yin Wang, Qingyong Hu, Jiaqi Tang, Yuting Zhang, Kaishen Yuan, Bin Huang, Zitong Yu, Dengbo He, Shuiguang Deng, Hao Chen, Yingcong Chen, Shiguang Shan

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

To Tell The Truth: Language of Deception and Language Models

Sanchaita Hazra, Bodhisattwa Prasad Majumder

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CL、cs.AI

Comments Accepted as a full paper in NAACL 2024 (Main)

Kiki or Bouba? Sound Symbolism in Vision-and-Language Models

Morris Alper, Hadar Averbuch-Elor

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted to NeurIPS 2023 (spotlight). Project webpage: https://kiki-bouba.github.io/

DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance

Zixuan Wang, Jia Jia, Shikun Sun, Haozhe Wu, Rong Han, Zhenyu Li, Di Tang, Jiaqing Zhou, Jiebo Luo

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.MM

Comments Accept to CVPR 2024

Large Scale Generative AI Text Applied to Sports and Music

Aaron Baughman, Stephen Hammer, Rahul Agarwal, Gozde Akay, Eduardo Morales, Tony Johnson, Leonid Karlinsky, Rogerio Feris

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 9 pages, 8 figures, 5 tables

Snapture -- A Novel Neural Architecture for Combined Static and Dynamic Hand Gesture Recognition

Hassan Ali, Doreen Jirak, Stefan Wermter

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments In Cognitive Computation(Accepted:30/06/2023, Published:17/07/2023),20 pages,20 figures,4 tables;Please find the published version/info to cite: https://doi.org/10.1007/s12559-023-10174-z;Repositories: https://zenodo.org/doi/10.5281/zenodo.10679196, https://zenodo.org/doi/10.5281/zenodo.10693816;This work was co-funded by Horizon Europe project TERAIS under Grant agreement number 101079338

Journal ref Cognitive Computation 15, 2014-2033 (2023)

SCRAPS: Speech Contrastive Representations of Acoustic and Phonetic Spaces

Ivan Vallés-Pérez, Grzegorz Beringer, Piotr Bilinski, Gary Cook, Roberto Barra-Chicote

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、eess.AS

Comments In proceedings of the 26th European Conference on Artificial Intelligence ECAI 2023. 8 pages + 1 appendix page

Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue

Guan-Ting Lin, Prashanth Gurunath Shivakumar, Ankur Gandhe, Chao-Han Huck Yang, Yile Gu, Shalini Ghosh, Andreas Stolcke, Hung-yi Lee, Ivan Bulyko

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted by ICASSP 2024. Camera-ready version

High-precision Voice Search Query Correction via Retrievable Speech-text Embedings

Christopher Li, Gary Wang, Kyle Kastner, Heng Su, Allen Chen, Andrew Rosenberg, Zhehuai Chen, Zelin Wu, Leonid Velikovich, Pat Rondon, Diamantino Caseiro, Petar Aleksic

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Detecting anxiety from short clips of free-form speech

Prabhat Agarwal, Akshat Jindal, Shreya Singh

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Listen As You Wish: Audio based Event Detection via Text-to-Audio Grounding in Smart Cities

Haoyu Tang, Yunxiao Wang, Jihua Zhu, Shuaike Zhang, Mingzhu Xu, Qinghai Zheng, Yupeng Hu

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.MM、eess.AS

Comments 16 pages

Speech and Text-Based Emotion Recognizer

Varun Sharma

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、cs.AI

Comments 11 pages 9 figures, 9 tables

On Robustness to Missing Video for Audiovisual Speech Recognition

Oscar Chang, Otavio Braga, Hank Liao, Dmitriy Serdyuk, Olivier Siohan

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、eess.AS

ControlLLM: Augment Language Models with Tools by Searching on Graphs

Zhaoyang Liu, Zeqiang Lai, Zhangwei Gao, Erfei Cui, Ziheng Li, Xizhou Zhu, Lewei Lu, Qifeng Chen, Yu Qiao, Jifeng Dai, Wenhai Wang

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.MM

Comments 24 pages, 9 figures, 12 tables

LiteVSR: Efficient Visual Speech Recognition by Learning from Speech Representations of Unlabeled Data

Hendrik Laux, Emil Mededovic, Ahmed Hallawa, Lukas Martin, Arne Peine, Anke Schmeink

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted for publication at ICASSP 2024

Seamless: Multilingual Expressive and Streaming Speech Translation

Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, John Hoffman, Min-Jae Hwang, Hirofumi Inaguma, Christopher Klaiber, Ilia Kulikov, Pengwei Li, Daniel Licht, Jean Maillard, Ruslan Mavlyutov, Alice Rakotoarison, Kaushik Ram Sadagopan, Abinesh Ramakrishnan, Tuan Tran, Guillaume Wenzek, Yilin Yang, Ethan Ye, Ivan Evtimov, Pierre Fernandez, Cynthia Gao, Prangthip Hansanti, Elahe Kalbassi, Amanda Kallet, Artyom Kozhevnikov, Gabriel Mejia Gonzalez, Robin San Roman, Christophe Touret, Corinne Wong, Carleigh Wood, Bokai Yu, Pierre Andrews, Can Balioglu, Peng-Jen Chen, Marta R. Costa-jussà, Maha Elbayad, Hongyu Gong, Francisco Guzmán, Kevin Heffernan, Somya Jain, Justine Kao, Ann Lee, Xutai Ma, Alex Mourachko, Benjamin Peloquin, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Anna Sun, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang, Mary Williamson

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applications

Karren D. Yang, Anurag Ranjan, Jen-Hao Rick Chang, Raviteja Vemulapalli, Oncel Tuzel

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

LIP-RTVE: An Audiovisual Database for Continuous Spanish in the Wild

David Gimeno-Gómez, Carlos-D. Martínez-Hinarejos

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted in Proceedings of LREC 2022 ( https://aclanthology.org/2022.lrec-1.294 )

Teach me with a Whisper: Enhancing Large Language Models for Analyzing Spoken Transcripts using Speech Embeddings

Fatema Hasan, Yulong Li, James Foulds, Shimei Pan, Bishwaranjan Bhattacharjee

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments 11 pages

Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and Audio

Xudong Xu, Dejan Markovic, Jacob Sandakly, Todd Keebler, Steven Krenn, Alexander Richard

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、eess.AS

Comments 37th Conference on Neural Information Processing Systems (NeurIPS 2023)

Proceedings of the 5th International Workshop on Reading Music Systems

Jorge Calvo-Zaragoza, Alexander Pacha, Elona Shatri

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、eess.AS

Comments Proceedings edited by Jorge Calvo-Zaragoza, Alexander Pacha and Elona Shatri