多模态大模型

AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs

Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, Mike Seltzer

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、cs.AI

GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing

Hao Lu, Xuesong Niu, Jiyao Wang, Yin Wang, Qingyong Hu, Jiaqi Tang, Yuting Zhang, Kaishen Yuan, Bin Huang, Zitong Yu, Dengbo He, Shuiguang Deng, Hao Chen, Yingcong Chen, Shiguang Shan

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

To Tell The Truth: Language of Deception and Language Models

Sanchaita Hazra, Bodhisattwa Prasad Majumder

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CL、cs.AI

Comments Accepted as a full paper in NAACL 2024 (Main)

Kiki or Bouba? Sound Symbolism in Vision-and-Language Models

Morris Alper, Hadar Averbuch-Elor

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted to NeurIPS 2023 (spotlight). Project webpage: https://kiki-bouba.github.io/

DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance

Zixuan Wang, Jia Jia, Shikun Sun, Haozhe Wu, Rong Han, Zhenyu Li, Di Tang, Jiaqing Zhou, Jiebo Luo

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.MM

Comments Accept to CVPR 2024

Large Scale Generative AI Text Applied to Sports and Music

Aaron Baughman, Stephen Hammer, Rahul Agarwal, Gozde Akay, Eduardo Morales, Tony Johnson, Leonid Karlinsky, Rogerio Feris

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 9 pages, 8 figures, 5 tables

Snapture -- A Novel Neural Architecture for Combined Static and Dynamic Hand Gesture Recognition

Hassan Ali, Doreen Jirak, Stefan Wermter

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments In Cognitive Computation(Accepted:30/06/2023, Published:17/07/2023),20 pages,20 figures,4 tables;Please find the published version/info to cite: https://doi.org/10.1007/s12559-023-10174-z;Repositories: https://zenodo.org/doi/10.5281/zenodo.10679196, https://zenodo.org/doi/10.5281/zenodo.10693816;This work was co-funded by Horizon Europe project TERAIS under Grant agreement number 101079338

Journal ref Cognitive Computation 15, 2014-2033 (2023)

SCRAPS: Speech Contrastive Representations of Acoustic and Phonetic Spaces

Ivan Vallés-Pérez, Grzegorz Beringer, Piotr Bilinski, Gary Cook, Roberto Barra-Chicote

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、eess.AS

Comments In proceedings of the 26th European Conference on Artificial Intelligence ECAI 2023. 8 pages + 1 appendix page

Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue

Guan-Ting Lin, Prashanth Gurunath Shivakumar, Ankur Gandhe, Chao-Han Huck Yang, Yile Gu, Shalini Ghosh, Andreas Stolcke, Hung-yi Lee, Ivan Bulyko

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted by ICASSP 2024. Camera-ready version

High-precision Voice Search Query Correction via Retrievable Speech-text Embedings

Christopher Li, Gary Wang, Kyle Kastner, Heng Su, Allen Chen, Andrew Rosenberg, Zhehuai Chen, Zelin Wu, Leonid Velikovich, Pat Rondon, Diamantino Caseiro, Petar Aleksic

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Detecting anxiety from short clips of free-form speech

Prabhat Agarwal, Akshat Jindal, Shreya Singh

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Listen As You Wish: Audio based Event Detection via Text-to-Audio Grounding in Smart Cities

Haoyu Tang, Yunxiao Wang, Jihua Zhu, Shuaike Zhang, Mingzhu Xu, Qinghai Zheng, Yupeng Hu

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.MM、eess.AS

Comments 16 pages

Speech and Text-Based Emotion Recognizer

Varun Sharma

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、cs.AI

Comments 11 pages 9 figures, 9 tables

On Robustness to Missing Video for Audiovisual Speech Recognition

Oscar Chang, Otavio Braga, Hank Liao, Dmitriy Serdyuk, Olivier Siohan

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、eess.AS

ControlLLM: Augment Language Models with Tools by Searching on Graphs

Zhaoyang Liu, Zeqiang Lai, Zhangwei Gao, Erfei Cui, Ziheng Li, Xizhou Zhu, Lewei Lu, Qifeng Chen, Yu Qiao, Jifeng Dai, Wenhai Wang

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.MM

Comments 24 pages, 9 figures, 12 tables

LiteVSR: Efficient Visual Speech Recognition by Learning from Speech Representations of Unlabeled Data

Hendrik Laux, Emil Mededovic, Ahmed Hallawa, Lukas Martin, Arne Peine, Anke Schmeink

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted for publication at ICASSP 2024

Seamless: Multilingual Expressive and Streaming Speech Translation

Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, John Hoffman, Min-Jae Hwang, Hirofumi Inaguma, Christopher Klaiber, Ilia Kulikov, Pengwei Li, Daniel Licht, Jean Maillard, Ruslan Mavlyutov, Alice Rakotoarison, Kaushik Ram Sadagopan, Abinesh Ramakrishnan, Tuan Tran, Guillaume Wenzek, Yilin Yang, Ethan Ye, Ivan Evtimov, Pierre Fernandez, Cynthia Gao, Prangthip Hansanti, Elahe Kalbassi, Amanda Kallet, Artyom Kozhevnikov, Gabriel Mejia Gonzalez, Robin San Roman, Christophe Touret, Corinne Wong, Carleigh Wood, Bokai Yu, Pierre Andrews, Can Balioglu, Peng-Jen Chen, Marta R. Costa-jussà, Maha Elbayad, Hongyu Gong, Francisco Guzmán, Kevin Heffernan, Somya Jain, Justine Kao, Ann Lee, Xutai Ma, Alex Mourachko, Benjamin Peloquin, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Anna Sun, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang, Mary Williamson

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applications

Karren D. Yang, Anurag Ranjan, Jen-Hao Rick Chang, Raviteja Vemulapalli, Oncel Tuzel

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

LIP-RTVE: An Audiovisual Database for Continuous Spanish in the Wild

David Gimeno-Gómez, Carlos-D. Martínez-Hinarejos

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted in Proceedings of LREC 2022 ( https://aclanthology.org/2022.lrec-1.294 )

Teach me with a Whisper: Enhancing Large Language Models for Analyzing Spoken Transcripts using Speech Embeddings

Fatema Hasan, Yulong Li, James Foulds, Shimei Pan, Bishwaranjan Bhattacharjee

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments 11 pages

Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and Audio

Xudong Xu, Dejan Markovic, Jacob Sandakly, Todd Keebler, Steven Krenn, Alexander Richard

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、eess.AS

Comments 37th Conference on Neural Information Processing Systems (NeurIPS 2023)

Proceedings of the 5th International Workshop on Reading Music Systems

Jorge Calvo-Zaragoza, Alexander Pacha, Elona Shatri

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、eess.AS

Comments Proceedings edited by Jorge Calvo-Zaragoza, Alexander Pacha and Elona Shatri

Generalized zero-shot audio-to-intent classification

Veera Raghavendra Elluru, Devang Kulshreshtha, Rohit Paturi, Sravan Bodapati, Srikanth Ronanki

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、eess.AS

CSLNSpeech: solving extended speech separation problem with the help of Chinese sign language

Jiasong Wu, Xuan Li, Taotao Li, Fanman Meng, Youyong Kong, Guanyu Yang, Lotfi Senhadji, Huazhong Shu

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments 13 pages, 6 figures, 5 tables

Automatic Disfluency Detection from Untranscribed Speech

Amrit Romana, Kazuhito Koishida, Emily Mower Provost

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、eess.AS

BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformer

Kunkun Pang, Dafei Qin, Yingruo Fan, Julian Habekost, Takaaki Shiratori, Junichi Yamagishi, Taku Komura

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Comments 12 pages, 13 figures

Vision-based Analysis of Driver Activity and Driving Performance Under the Influence of Alcohol

Ross Greer, Akshay Gopalkrishnan, Sumega Mandadi, Pujitha Gunaratne, Mohan M. Trivedi, Thomas D. Marcotte

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Withdrawn at the request of industry research collaborators, per contract agreement

A Comparative Study of Voice Conversion Models with Large-Scale Speech and Singing Data: The T13 Systems for the Singing Voice Conversion Challenge 2023

Ryuichi Yamamoto, Reo Yoneyama, Lester Phillip Violeta, Wen-Chin Huang, Tomoki Toda

专题命中 音频语音多模态 ：any-to-any(abstract)；分类 cs.CL、eess.AS

Comments Accepted to ASRU 2023

Audio Visual Speaker Localization from EgoCentric Views

Jinzheng Zhao, Yong Xu, Xinyuan Qian, Wenwu Wang

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.MM、eess.AS

AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model

Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, Kavya Srinet, Babak Damavandi, Anuj Kumar

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL