多模态大模型

Detecting Children with Autism Spectrum Disorder based on Script-Centric Behavior Understanding with Emotional Enhancement

Wenxing Liu, Yueran Pan, Dong Zhang, Hongzhu Deng, Xiaobing Zou, Ming Li

机构 * School of Computer Science, Wuhan University（武汉大学计算机学院） ； Digital Innovation Research Center, Duke Kunshan University（杜克昆山大学数字创新研究中心） ； School of Electronics and Information Technology, Sun Yat-sen University（中山大学电子与信息学院） ； Child Development and Behavior Center, Third Affiliated Hospital of Sun Yat-sen University（中山大学第三附属医院儿童发展与行为中心） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI

Comments 15 pages, 12 figures, sumbitted to IEEE transactions on affective computing

Qieemo: Speech Is All You Need in the Emotion Recognition in Conversations

Jinming Chen, Jingyi Fang, Yuanzhong Zheng, Yaoxuan Wang, Haojun Fei

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.AI、eess.AS

DeepSound-V1: Start to Think Step-by-Step in the Audio Generation from Videos

Yunming Liang, Zihao Chen, Chaofan Ding, Xinhan Di

专题命中 音频语音多模态 ：multi-modal(abstract)；MLLM(abstract)；分类 cs.CV、eess.AS

Comments 11 pages, 6 figures

Seeing Speech and Sound: Distinguishing and Locating Audios in Visual Scenes

Hyeonggon Ryu, Seongyu Kim, Joon Son Chung, Arda Senocak

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments CVPR 2025

Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics

Chen Liu, Liying Yang, Peike Li, Dadong Wang, Lincheng Li, Xin Yu

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted by CVPR2025

Seeing Sound: Assembling Sounds from Visuals for Audio-to-Image Generation

Darius Petermann, Mahdi M. Kalayeh

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Linguistics-Vision Monotonic Consistent Network for Sign Language Production

Xu Wang, Shengeng Tang, Peipei Song, Shuo Wang, Dan Guo, Richang Hong

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted by ICASSP 2025

A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation

Louis Airale, Dominique Vaufreydaz, Xavier Alameda-Pineda

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Large-scale unsupervised audio pre-training for video-to-speech synthesis

Triantafyllos Kefalas, Yannis Panagakis, Maja Pantic

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Corrected typos. This work has been submitted to the IEEE for possible publication

Speech Recognition Rescoring with Large Speech-Text Foundation Models

Prashanth Gurunath Shivakumar, Jari Kolehmainen, Aditya Gourav, Yi Gu, Ankur Gandhe, Ariya Rastrow, Ivan Bulyko

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation

Se Jin Park, Chae Won Kim, Hyeongseop Rha, Minsu Kim, Joanna Hong, Jeong Hun Yeo, Yong Man Ro

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI

Comments Accepted to ACL 2024 (Oral)

Generative Expressive Conversational Speech Synthesis

Rui Liu, Yifan Hu, Yi Ren, Xiang Yin, Haizhou Li

专题命中 音频语音多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CL、eess.AS

Comments 14 pages, 6 figures, 8 tables. Accepted by ACM MM 2024

Face-voice Association in Multilingual Environments (FAME) Challenge 2024 Evaluation Plan

Muhammad Saad Saeed, Shah Nawaz, Muhammad Salman Tahir, Rohan Kumar Das, Muhammad Zaigham Zaheer, Marta Moscati, Markus Schedl, Muhammad Haris Khan, Karthik Nandakumar, Muhammad Haroon Yousaf

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments ACM Multimedia Conference - Grand Challenge

An End-to-End Speech Summarization Using Large Language Model

Hengchao Shang, Zongyao Li, Jiaxin Guo, Shaojun Li, Zhiqiang Rao, Yuanchang Luo, Daimeng Wei, Hao Yang

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Comments InterSpeech 2024

WikiMuTe: A web-sourced dataset of semantic descriptions for music audio

Benno Weck, Holger Kirchhoff, Peter Grosche, Xavier Serra

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Comments Submitted to 30th International Conference on MultiMedia Modeling (MMM2024). This preprint has not undergone peer review or any post-submission improvements or corrections

Journal ref The Version of Record of this contribution is published in MultiMedia Modeling. MMM 2024. Lecture Notes in Computer Science, vol 14565. Springer, Cham

SALMONN: Towards Generic Hearing Abilities for Large Language Models

Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Chao Zhang

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Speech Emotion Recognition with Distilled Prosodic and Linguistic Affect Representations

Debaditya Shome, Ali Etemad

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI

Comments Accepted at ICASSP 2024

Novel-View Acoustic Synthesis

Changan Chen, Alexander Richard, Roman Shapovalov, Vamsi Krishna Ithapu, Natalia Neverova, Kristen Grauman, Andrea Vedaldi

专题命中 音频语音多模态 ：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted at CVPR 2023. Project page: https://vision.cs.utexas.edu/projects/nvas

Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation

Yiyang Su, Ali Vosoughi, Shijian Deng, Yapeng Tian, Chenliang Xu

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted at ICCV 2023 - AV4D, 4 figures, 3 tables

Tuning Large language model for End-to-end Speech Translation

Hao Zhang, Nianwen Si, Yaqi Chen, Wenlin Zhang, Xukui Yang, Dan Qu, Xiaolin Jiao

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Music- and Lyrics-driven Dance Synthesis

Wenjie Yin, Qingyuan Yao, Yi Yu, Hang Yin, Danica Kragic, Mårten Björkman

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.MM、eess.AS

LLaSM: Large Language and Speech Model

Yu Shu, Siwei Dong, Guangyao Chen, Wenhao Huang, Ruihua Zhang, Daochen Shi, Qiqi Xiang, Yemin Shi

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Exploring Efficient-Tuned Learning Audio Representation Method from BriVL

Sen Fang, Yangjian Wu, Bowen Gao, Jingwen Cai, Teik Toe Teoh

专题命中 音频语音多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.AI、eess.AS

Comments 13 pages, 2023.3 Finished

Exploring the Role of Audio in Video Captioning

Yuhan Shen, Linjie Yang, Longyin Wen, Haichao Yu, Ehsan Elhamifar, Heng Wang

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Improving speech translation by fusing speech and text

Wenbiao Yin, Zhicheng Liu, Chengqi Zhao, Tao Wang, Jian Tong, Rong Ye

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

A Light Weight Model for Active Speaker Detection

Junhua Liao, Haihan Duan, Kanghui Feng, Wanbing Zhao, Yanbing Yang, Liangyin Chen

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted by CVPR 2023

Object Segmentation with Audio Context

Kaihui Zheng, Yuqing Ren, Zixin Shen, Tianxu Qin

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Research project for Introduction to Deep Learning (11785) at Carnegie Mellon University

An Analysis of Semantically-Aligned Speech-Text Embeddings

Muhammad Huzaifah, Ivan Kukanov

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Comments This is the accepted version of the paper published at IEEE Spoken Language Technology (SLT) Workshop 2022

Contrastive Audio-Language Learning for Music

Ilaria Manco, Emmanouil Benetos, Elio Quinton, György Fazekas

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Comments Accepted to ISMIR 2022

Is Lip Region-of-Interest Sufficient for Lipreading?

Jing-Xuan Zhang, Gen-Shun Wan, Jia Pan

专题命中 音频语音多模态 ：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments preprint