多模态大模型

Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech Generators

Wiebke Hutiri, Oresiti Papakyriakopoulos, Alice Xiang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 17 pages, 4 tables, 4 figures Accepted at the 2024 ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT '24)

Visual and audio scene classification for detecting discrepancies in video: a baseline method and experimental protocol

Konstantinos Apostolidis, Jakob Abesser, Luca Cuccovillo, Vasileios Mezaris

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted for publication, 3rd ACM Int. Workshop on Multimedia AI against Disinformation (MAD'24) at ACM ICMR'24, June 10, 2024, Phuket, Thailand. This is the "accepted version"

Synthesizing Audio from Silent Video using Sequence to Sequence Modeling

Hugo Garrido-Lestache Belinchon, Helina Mulugeta, Adam Haile

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.AI、eess.AS

Modal-adaptive Knowledge-enhanced Graph-based Financial Prediction from Monetary Policy Conference Calls with LLM

Kun Ouyang, Yi Liu, Shicheng Li, Ruihan Bao, Keiko Harimoto, Xu Sun

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)

Comments Accepted by LREC Coling 2024 -FinNLP (oral)

Audio is all in one: speech-driven gesture synthetics using WavLM pre-trained model

Fan Zhang, Naye Ji, Fuxing Gao, Siyuan Zhao, Zhaohan Wang, Shunman Li

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments This article needs major revision

Robust Active Speaker Detection in Noisy Environments

Siva Sai Nagender Vasireddy, Chenxu Zhang, Xiaohu Guo, Yapeng Tian

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 15 pages, 5 figures

JEP-KD: Joint-Embedding Predictive Architecture Based Knowledge Distillation for Visual Speech Recognition

Chang Sun, Hong Yang, Bo Qin

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、eess.AS

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Fan Zhang, Zhaohan Wang, Xin Lyu, Siyuan Zhao, Mengjian Li, Weidong Geng, Naye Ji, Hui Du, Fuxing Gao, Hao Wu, Shunman Li

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments 12 pages,

LM2D: Lyrics- and Music-Driven Dance Synthesis

Wenjie Yin, Xuejiao Zhao, Yi Yu, Hang Yin, Danica Kragic, Mårten Björkman

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

MOSAIC: Learning Unified Multi-Sensory Object Property Representations for Robot Learning via Interactive Perception

Gyan Tatiya, Jonathan Francis, Ho-Hsiang Wu, Yonatan Bisk, Jivko Sinapov

专题命中 音频语音多模态 ：multimodal(abstract)；multimodal foundation model(abstract)

Comments Accepted to the 2024 IEEE International Conference on Robotics and Automation (ICRA), May 13 to 17, 2024; Yokohama, Japan

Effectiveness of Text, Acoustic, and Lattice-based representations in Spoken Language Understanding tasks

Esaú Villatoro-Tello, Srikanth Madikeri, Juan Zuluaga-Gomez, Bidisha Sharma, Seyyed Saeed Sarfjoo, Iuliia Nigmatulina, Petr Motlicek, Alexei V. Ivanov, Aravind Ganapathiraju

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted in ICASSP 2023

Journal ref ICASSP 2023

Improved Cross-Lingual Transfer Learning For Automatic Speech Translation

Sameer Khurana, Nauman Dawalatabad, Antoine Laurent, Luis Vicente, Pablo Gimeno, Victoria Mingote, James Glass

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Detecting Check-Worthy Claims in Political Debates, Speeches, and Interviews Using Audio Data

Petar Ivanov, Ivan Koychev, Momchil Hardalov, Preslav Nakov

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Check-Worthiness, Fact-Checking, Fake News, Misinformation, Disinformation, Political Debates, Multimodality

Journal ref ICASSP 2024

Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper

Jeong Hun Yeo, Minsu Kim, Shinji Watanabe, Yong Man Ro

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Accepted at ICASSP 2024

Diffusion Models as Masked Audio-Video Learners

Elvis Nunez, Yanzi Jin, Mohammad Rastegari, Sachin Mehta, Maxwell Horton

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Camera-ready version for the Machine Learning for Audio Workshop at NeurIPS 2023

Rethinking and Improving Multi-task Learning for End-to-end Speech Translation

Yuhao Zhang, Chen Xu, Bei Li, Hao Chen, Tong Xiao, Chunliang Zhang, Jingbo Zhu

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted to EMNLP2023 main conference

Toward Joint Language Modeling for Speech Units and Text

Ju-Chieh Chou, Chung-Ming Chien, Wei-Ning Hsu, Karen Livescu, Arun Babu, Alexis Conneau, Alexei Baevski, Michael Auli

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments EMNLP findings 2023

DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation

Qingkai Fang, Yan Zhou, Yang Feng

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments NeurIPS 2023. Audio samples are available at https://ictnlp.github.io/daspeech-demo/

Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis

Shivam Mehta, Siyang Wang, Simon Alexanderson, Jonas Beskow, Éva Székely, Gustav Eje Henter

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments 7 pages, 2 figures, presented at the ISCA Speech Synthesis Workshop (SSW) 2023

Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization

Puyuan Peng, Brian Yan, Shinji Watanabe, David Harwath

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Interspeech 2023

Improving Joint Speech-Text Representations Without Alignment

Cal Peyser, Zhong Meng, Ke Hu, Rohit Prabhavalkar, Andrew Rosenberg, Tara N. Sainath, Michael Picheny, Kyunghyun Cho

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Journal ref INTERSPEECH 2023

Learning Music-Dance Representations through Explicit-Implicit Rhythm Synchronization

Jiashuo Yu, Junfu Pu, Ying Cheng, Rui Feng, Ying Shan

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted for publication in IEEE Transactions on Multimedia

MAViL: Masked Audio-Video Learners

Po-Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Haoqi Fan, Yanghao Li, Shang-Wen Li, Gargi Ghosh, Jitendra Malik, Christoph Feichtenhofer

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Technical report

AudioPaLM: A Large Language Model That Can Speak and Listen

Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, Hannah Muckenhirn, Dirk Padfield, James Qin, Danny Rozenberg, Tara Sainath, Johan Schalkwyk, Matt Sharifi, Michelle Tadmor Ramanovich, Marco Tagliasacchi, Alexandru Tudor, Mihajlo Velimirović, Damien Vincent, Jiahui Yu, Yongqiang Wang, Vicky Zayats, Neil Zeghidour, Yu Zhang, Zhishuai Zhang, Lukas Zilka, Christian Frank

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Technical report

SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data

Ziqiang Zhang, Sanyuan Chen, Long Zhou, Yu Wu, Shuo Ren, Shujie Liu, Zhuoyuan Yao, Xun Gong, Lirong Dai, Jinyu Li, Furu Wei

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments We have corrected the errors in the pre-training data for SpeechLM-P Base models, new results are updated

A Novel Interpretable and Generalizable Re-synchronization Model for Cued Speech based on a Multi-Cuer Corpus

Lufei Gao, Shan Huang, Li Liu

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments 5 pages, 4 figures, Accepted to INTERSPEECH2023

Multi-Scale Attention for Audio Question Answering

Guangyao Li, Yixin Xu, Di Hu

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments Accepted by InterSpeech 2023

Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation

Minglun Han, Feilong Chen, Jing Shi, Shuang Xu, Bo Xu

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted by INTERSPEECH 2023

CIF-PT: Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Training

Linhao Dong, Zhecheng An, Peihao Wu, Jun Zhang, Lu Lu, Zejun Ma

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted by ACL 2023 Findings

i-Code Studio: A Configurable and Composable Framework for Integrative AI

Yuwei Fang, Mahmoud Khademi, Chenguang Zhu, Ziyi Yang, Reid Pryzant, Yichong Xu, Yao Qian, Takuya Yoshioka, Lu Yuan, Michael Zeng, Xuedong Huang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI