多模态大模型

Survey of Design Paradigms for Social Robots

Rita Frieske, Xiaoyu Mo, Yini Fang, Jay Nieles, Bertram E. Shi

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL

The Future is Meta: Metadata, Formats and Perspectives towards Interactive and Personalized AV Content

Alexander Weller, Werner Bleisteiner, Christian Hufnagel, Michael Iber

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.MM

Comments 12 pages, 4 figures, submitted to the Tonmeistertagung 32

E-chat: Emotion-sensitive Spoken Dialogue System with Large Language Models

Hongfei Xue, Yuhao Liang, Bingshen Mu, Shiliang Zhang, Mengzhe Chen, Qian Chen, Lei Xie

专题命中 音频语音多模态 ：multimodal(abstract)；分类 eess.AS

Comments 5 pages, 3 figures

Multi-Stage Face-Voice Association Learning with Keynote Speaker Diarization

Ruijie Tao, Zhan Shi, Yidi Jiang, Duc-Tuan Truong, Eng-Siong Chng, Massimo Alioto, Haizhou Li

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 eess.AS

Unveiling and Mitigating Bias in Audio Visual Segmentation

Peiwen Sun, Honggang Zhang, Di Hu

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV

Comments Accepted by ACM MM 24 (ORAL)

Seal: Advancing Speech Language Models to be Few-Shot Learners

Shuyu Lei, Lingen Liu, Jiaolong Yang, Yasen Jiao, Yuxiang Yang, Yushu Yang, Xiang Guo

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL

DiveSound: LLM-Assisted Automatic Taxonomy Construction for Diverse Audio Generation

Baihan Li, Zeyu Xie, Xuenan Xu, Yiwei Guo, Ming Yan, Ji Zhang, Kai Yu, Mengyue Wu

专题命中 音频语音多模态 ：multimodal(abstract)；分类 eess.AS

Towards Weakly Supervised Text-to-Audio Grounding

Xuenan Xu, Ziyang Ma, Mengyue Wu, Kai Yu

专题命中 音频语音多模态 ：multimodal(abstract)；分类 eess.AS

Can Textual Semantics Mitigate Sounding Object Segmentation Preference?

Yaoting Wang, Peiwen Sun, Yuanchao Li, Honggang Zhang, Di Hu

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV

Comments Accepted by ECCV2024

Ethics of Generating Synthetic MRI Vocal Tract Views from the Face

Muhammad Suhaib Shahid, Gleb E. Yakubov, Andrew P. French

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV

Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data

Nahema Marchal, Rachel Xu, Rasmi Elasmar, Iason Gabriel, Beth Goldberg, William Isaac

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI

SRC4VC: Smartphone-Recorded Corpus for Voice Conversion Benchmark

Yuki Saito, Takuto Igarashi, Kentaro Seki, Shinnosuke Takamichi, Ryuichi Yamamoto, Kentaro Tachibana, Hiroshi Saruwatari

专题命中 音频语音多模态 ：any-to-any(abstract)；分类 eess.AS

Comments Accepted for INTERSPEECH 2024, corpus project page: https://y-saito.sakura.ne.jp/sython/Corpus/SRC4VC/index.html

Zero-Shot Audio Captioning Using Soft and Hard Prompts

Yiming Zhang, Xuenan Xu, Ruoyi Du, Haohe Liu, Yuan Dong, Zheng-Hua Tan, Wenwu Wang, Zhanyu Ma

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 eess.AS

Comments Submitted to IEEE/ACM Transactions on Audio, Speech and Language Processing

G-VOILA: Gaze-Facilitated Information Querying in Daily Scenarios

Zeyu Wang, Yuanchun Shi, Yuntao Wang, Yuchen Yao, Kun Yan, Yuhan Wang, Lei Ji, Xuhai Xu, Chun Yu

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI

Comments 25 pages, 12 figures

NeRFFaceSpeech: One-shot Audio-driven 3D Talking Head Synthesis via Generative Prior

Gihoon Kim, Kwanggyoon Seo, Sihun Cha, Junyong Noh

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV

Comments 11 pages, 5 figures

Audio Matters Too! Enhancing Markerless Motion Capture with Audio Signals for String Performance Capture

Yitong Jin, Zhiping Qiu, Yi Shi, Shuangpeng Sun, Chongwu Wang, Donghao Pan, Jiachen Zhao, Zhenghao Liang, Yuan Wang, Xiaobing Li, Feng Yu, Tao Yu, Qionghai Dai

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.MM

Comments SIGGRAPH2024

LLM-AD: Large Language Model based Audio Description System

Peng Chu, Jiang Wang, Andre Abrantes

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV

Learn2Talk: 3D Talking Face Learns from 2D Talking Face

Yixiang Zhuang, Baoping Cheng, Yao Cheng, Yuntao Jin, Renshuai Liu, Chengyang Li, Xuan Cheng, Jing Liao, Juncong Lin

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV

LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models

Zihan Zhao, Yiyang Jiang, Heyang Liu, Yanfeng Wang, Yu Wang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL

Vivo : une approche multimodale de la synthese concatenative par corpus dans le cadre d'une oeuvre audiovisuelle immersive

Mateo Fayet

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 eess.AS

Comments in French language

Leveraging Interesting Facts to Enhance User Engagement with Conversational Interfaces

Nikhita Vedula, Giuseppe Castellucci, Eugene Agichtein, Oleg Rokhlenko, Shervin Malmasi

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL

Comments 10 pages, 1 figure

Task Agnostic Architecture for Algorithm Induction via Implicit Composition

Sahil J. Sindhi, Ignas Budvytis

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.AI

Comments 12 pages, 2 figures, 2024 ICLR Generative Models for Decision Making Workshop

BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognition

Alexandros Haliassos, Andreas Zinonos, Rodrigo Mira, Stavros Petridis, Maja Pantic

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV

Comments ICASSP 2024. Code: https://github.com/ahaliassos/raven

Composite Backdoor Attacks Against Large Language Models

Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, Yang Zhang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL

Comments To Appear in Findings of the Association for Computational Linguistics: NAACL 2024, June 2024

Target Speech Extraction with Pre-trained AV-HuBERT and Mask-And-Recover Strategy

Wenxuan Wu, Xueyuan Chen, Xixin Wu, Haizhou Li, Helen Meng

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 eess.AS

Comments Accepted by IJCNN 2024

AFL-Net: Integrating Audio, Facial, and Lip Modalities with a Two-step Cross-attention for Robust Speaker Diarization in the Wild

Yongkang Yin, Xu Li, Ying Shan, Yuexian Zou

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.MM

Mobile Foundation Model as Firmware

Jinliang Yuan, Chen Yang, Dongqi Cai, Shihe Wang, Xin Yuan, Zeling Zhang, Xiang Li, Dingge Zhang, Hanzi Mei, Xianqing Jia, Shangguang Wang, Mengwei Xu

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI

Comments 17 pages, 15 figures, published to ACM MobiCom'24

Journal ref The 30th Annual International Conference on Mobile Computing and Networking, 2024

MuseChat: A Conversational Music Recommendation System for Videos

Zhikang Dong, Bin Chen, Xiulong Liu, Pawel Polak, Peng Zhang

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV

A Detailed Audio-Text Data Simulation Pipeline using Single-Event Sounds

Xuenan Xu, Xiaohang Xu, Zeyu Xie, Pingyue Zhang, Mengyue Wu, Kai Yu

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 eess.AS

BLAT: Bootstrapping Language-Audio Pre-training based on AudioSet Tag-guided Synthetic Data

Xuenan Xu, Zhiling Zhang, Zelin Zhou, Pingyue Zhang, Zeyu Xie, Mengyue Wu, Kenny Q. Zhu

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 eess.AS