多模态大模型 - arXivDaily 专题

2311.02248 2024-06-17 cs.CL cs.AI eess.AS 67%

COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning

Jing Pan, Jian Wu, Yashesh Gaur, Sunit Sivasankaran, Zhuo Chen, Shujie Liu, Jinyu Li

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

2406.05629 2024-06-11 cs.CV cs.CL cs.IR cs.LG cs.SD eess.AS 67%

Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Language

Mark Hamilton, Andrew Zisserman, John R. Hershey, William T. Freeman

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Computer Vision and Pattern Recognition 2024

2405.11093 2024-06-10 eess.AS cs.CL cs.MM cs.SD 67%

AudioSetMix: Enhancing Audio-Language Datasets with LLM-Assisted Augmentations

David Xu

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments typos corrected

2406.04432 2024-06-10 eess.AS cs.AI cs.CL 67%

LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognition

Sreyan Ghosh, Sonal Kumar, Ashish Seth, Purva Chiniya, Utkarsh Tyagi, Ramani Duraiswami, Dinesh Manocha

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments InterSpeech 2024. Code and Data: https://github.com/Sreyan88/LipGER

2406.01624 2024-06-07 eess.AS cs.AI cs.CL cs.LG cs.SD 67%

Unveiling Hidden Factors: Explainable AI for Feature Boosting in Speech Emotion Recognition

Alaa Nfissi, Wassim Bouachir, Nizar Bouguila, Brian Mishara

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Published in: Springer Nature International Journal of Applied Intelligence (2024)

Journal ref Applied Intelligence (2024), 1-24

2405.19103 2024-05-30 cs.CR cs.LG 67%

Voice Jailbreak Attacks Against GPT-4o

Xinyue Shen, Yixin Wu, Michael Backes, Yang Zhang

专题命中音频语音多模态：multimodal(abstract)；MLLM(abstract)

2405.10272 2024-05-17 cs.CV cs.AI cs.SD eess.AS eess.IV 67%

Faces that Speak: Jointly Synthesising Talking Face and Speech from Text

Youngjoon Jang, Ji-Hoon Kim, Junseok Ahn, Doyeop Kwak, Hong-Sun Yang, Yoon-Cheol Ju, Il-Hwan Kim, Byeong-Yeol Kim, Joon Son Chung

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments CVPR 2024

2405.10025 2024-05-17 cs.CL cs.AI cs.LG cs.SD eess.AS 67%

Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models

Yuchen Hu, Chen Chen, Chengwei Qin, Qiushi Zhu, Eng Siong Chng, Ruizhe Li

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 14 pages, Accepted by ACL 2024

2402.01708 2024-05-16 cs.CL cs.AI cs.CY eess.AS 67%

Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech Generators

Wiebke Hutiri, Oresiti Papakyriakopoulos, Alice Xiang

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 17 pages, 4 tables, 4 figures Accepted at the 2024 ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT '24)

2405.00384 2024-05-02 cs.CV cs.MM cs.SD eess.AS 67%

Visual and audio scene classification for detecting discrepancies in video: a baseline method and experimental protocol

Konstantinos Apostolidis, Jakob Abesser, Luca Cuccovillo, Vasileios Mezaris

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted for publication, 3rd ACM Int. Workshop on Multimedia AI against Disinformation (MAD'24) at ACM ICMR'24, June 10, 2024, Phuket, Thailand. This is the "accepted version"

2404.17608 2024-04-30 cs.SD cs.AI cs.CV cs.LG eess.AS 67%

Synthesizing Audio from Silent Video using Sequence to Sequence Modeling

Hugo Garrido-Lestache Belinchon, Helina Mulugeta, Adam Haile

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.AI、eess.AS

2403.16055 2024-04-23 cs.CE 67%

Modal-adaptive Knowledge-enhanced Graph-based Financial Prediction from Monetary Policy Conference Calls with LLM

Kun Ouyang, Yi Liu, Shicheng Li, Ruihan Bao, Keiko Harimoto, Xu Sun

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)

Comments Accepted by LREC Coling 2024 -FinNLP (oral)

2308.05995 2024-04-16 cs.SD cs.AI cs.GR cs.MM eess.AS 67%

Audio is all in one: speech-driven gesture synthetics using WavLM pre-trained model

Fan Zhang, Naye Ji, Fuxing Gao, Siyuan Zhao, Zhaohan Wang, Shunman Li

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments This article needs major revision

2403.19002 2024-04-02 cs.MM cs.CV cs.SD eess.AS 67%

Robust Active Speaker Detection in Noisy Environments

Siva Sai Nagender Vasireddy, Chenxu Zhang, Xiaohu Guo, Yapeng Tian

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 15 pages, 5 figures

2403.18843 2024-03-29 cs.CV cs.CL cs.LG cs.SD eess.AS 67%

JEP-KD: Joint-Embedding Predictive Architecture Based Knowledge Distillation for Visual Speech Recognition

Chang Sun, Hong Yang, Bo Qin

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.CL、eess.AS

2403.10805 2024-03-19 cs.SD cs.AI cs.CV cs.GR cs.HC eess.AS 67%

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Fan Zhang, Zhaohan Wang, Xin Lyu, Siyuan Zhao, Mengjian Li, Weidong Geng, Naye Ji, Hui Du, Fuxing Gao, Hao Wu, Shunman Li

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments 12 pages,

2403.09407 2024-03-15 cs.SD cs.AI cs.LG cs.MM eess.AS 67%

LM2D: Lyrics- and Music-Driven Dance Synthesis

Wenjie Yin, Xuejiao Zhao, Yi Yu, Hang Yin, Danica Kragic, Mårten Björkman

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

2309.08508 2024-02-26 cs.RO 67%

MOSAIC: Learning Unified Multi-Sensory Object Property Representations for Robot Learning via Interactive Perception

Gyan Tatiya, Jonathan Francis, Ho-Hsiang Wu, Yonatan Bisk, Jivko Sinapov

专题命中音频语音多模态：multimodal(abstract)；multimodal foundation model(abstract)

Comments Accepted to the 2024 IEEE International Conference on Robotics and Automation (ICRA), May 13 to 17, 2024; Yokohama, Japan

2212.08489 2024-02-02 cs.CL cs.AI cs.SD eess.AS 67%

Effectiveness of Text, Acoustic, and Lattice-based representations in Spoken Language Understanding tasks

Esaú Villatoro-Tello, Srikanth Madikeri, Juan Zuluaga-Gomez, Bidisha Sharma, Seyyed Saeed Sarfjoo, Iuliia Nigmatulina, Petr Motlicek, Alexei V. Ivanov, Aravind Ganapathiraju

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted in ICASSP 2023

Journal ref ICASSP 2023

2306.00789 2024-01-26 cs.CL cs.AI eess.AS eess.SP 67%

Improved Cross-Lingual Transfer Learning For Automatic Speech Translation

Sameer Khurana, Nauman Dawalatabad, Antoine Laurent, Luis Vicente, Pablo Gimeno, Victoria Mingote, James Glass

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

2306.05535 2024-01-19 cs.CL cs.AI cs.IR cs.LG cs.SD eess.AS 67%

Detecting Check-Worthy Claims in Political Debates, Speeches, and Interviews Using Audio Data

Petar Ivanov, Ivan Koychev, Momchil Hardalov, Preslav Nakov

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Check-Worthiness, Fact-Checking, Fake News, Misinformation, Disinformation, Political Debates, Multimodality

Journal ref ICASSP 2024

2309.08535 2024-01-15 cs.CV cs.AI eess.AS 67%

Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper

Jeong Hun Yeo, Minsu Kim, Shinji Watanabe, Yong Man Ro

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Accepted at ICASSP 2024

2310.03937 2024-01-08 cs.SD cs.CV cs.MM eess.AS 67%

Diffusion Models as Masked Audio-Video Learners

Elvis Nunez, Yanzi Jin, Mohammad Rastegari, Sachin Mehta, Maxwell Horton

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Camera-ready version for the Machine Learning for Audio Workshop at NeurIPS 2023

2311.03810 2023-11-08 cs.CL cs.AI cs.SD eess.AS 67%

Rethinking and Improving Multi-task Learning for End-to-end Speech Translation

Yuhao Zhang, Chen Xu, Bei Li, Hao Chen, Tong Xiao, Chunliang Zhang, Jingbo Zhu

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted to EMNLP2023 main conference

2310.08715 2023-10-16 cs.CL cs.AI cs.SD eess.AS 67%

Toward Joint Language Modeling for Speech Units and Text

Ju-Chieh Chou, Chung-Ming Chien, Wei-Ning Hsu, Karen Livescu, Arun Babu, Alexis Conneau, Alexei Baevski, Michael Auli

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments EMNLP findings 2023

2310.07403 2023-10-12 cs.CL cs.AI cs.SD eess.AS 67%

DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation

Qingkai Fang, Yan Zhou, Yang Feng

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments NeurIPS 2023. Audio samples are available at https://ictnlp.github.io/daspeech-demo/

2306.09417 2023-09-15 eess.AS cs.AI cs.CV cs.HC cs.LG 67%

Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis

Shivam Mehta, Siyang Wang, Simon Alexanderson, Jonas Beskow, Éva Székely, Gustav Eje Henter

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments 7 pages, 2 figures, presented at the ISCA Speech Synthesis Workshop (SSW) 2023

2305.11095 2023-08-17 eess.AS cs.AI cs.CL cs.LG cs.SD 67%

Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization

Puyuan Peng, Brian Yan, Shinji Watanabe, David Harwath

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Interspeech 2023

2308.06125 2023-08-14 cs.CL cs.AI cs.SD eess.AS 67%

Improving Joint Speech-Text Representations Without Alignment

Cal Peyser, Zhong Meng, Ke Hu, Rohit Prabhavalkar, Andrew Rosenberg, Tara N. Sainath, Michael Picheny, Kyunghyun Cho

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Journal ref INTERSPEECH 2023

2207.03190 2023-08-11 cs.SD cs.CV cs.MM eess.AS 67%

Learning Music-Dance Representations through Explicit-Implicit Rhythm Synchronization

Jiashuo Yu, Junfu Pu, Ying Cheng, Rui Feng, Ying Shan

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted for publication in IEEE Transactions on Multimedia

1. 音频语音多模态 4597 篇

COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning

Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Language

AudioSetMix: Enhancing Audio-Language Datasets with LLM-Assisted Augmentations

LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognition

Unveiling Hidden Factors: Explainable AI for Feature Boosting in Speech Emotion Recognition

Voice Jailbreak Attacks Against GPT-4o

Faces that Speak: Jointly Synthesising Talking Face and Speech from Text

Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models

Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech Generators

Visual and audio scene classification for detecting discrepancies in video: a baseline method and experimental protocol

Synthesizing Audio from Silent Video using Sequence to Sequence Modeling

Modal-adaptive Knowledge-enhanced Graph-based Financial Prediction from Monetary Policy Conference Calls with LLM

Audio is all in one: speech-driven gesture synthetics using WavLM pre-trained model

Robust Active Speaker Detection in Noisy Environments

JEP-KD: Joint-Embedding Predictive Architecture Based Knowledge Distillation for Visual Speech Recognition

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

LM2D: Lyrics- and Music-Driven Dance Synthesis

MOSAIC: Learning Unified Multi-Sensory Object Property Representations for Robot Learning via Interactive Perception

Effectiveness of Text, Acoustic, and Lattice-based representations in Spoken Language Understanding tasks

Improved Cross-Lingual Transfer Learning For Automatic Speech Translation

Detecting Check-Worthy Claims in Political Debates, Speeches, and Interviews Using Audio Data

Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper

Diffusion Models as Masked Audio-Video Learners

Rethinking and Improving Multi-task Learning for End-to-end Speech Translation

Toward Joint Language Modeling for Speech Units and Text

DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation

Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis

Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization

Improving Joint Speech-Text Representations Without Alignment

Learning Music-Dance Representations through Explicit-Implicit Rhythm Synchronization