多模态大模型

A Cross-Modal Approach to Silent Speech with LLM-Enhanced Recognition

Tyler Benster, Guy Wilson, Reshef Elisha, Francis R Willett, Shaul Druckmann

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.AI、eess.AS

Evaluation of Audio-Visual Alignments in Visually Grounded Speech Models

Khazar Khorrami, Okko Räsänen

专题命中 音频语音多模态 ：audio-visual(title)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments To be published in Proc. Interspeech-2021, Brno, Czech Republic

A multimodal dynamical variational autoencoder for audiovisual speech representation learning

Samir Sadok, Simon Leglaive, Laurent Girin, Xavier Alameda-Pineda, Renaud Séguier

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.MM、eess.AS

Comments 14 figures, https://samsad35.github.io/site-mdvae/

Computation and Parameter Efficient Multi-Modal Fusion Transformer for Cued Speech Recognition

Lei Liu, Li Liu, Haizhou Li

专题命中 音频语音多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、eess.AS

Comments Accepted by TASLP

Leveraging the Video-level Semantic Consistency of Event for Audio-visual Event Localization

Yuanyuan Jiang, Jianqin Yin, Yonghao Dang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments 13 pages, 10 figures, Accepted by IEEE Transactions on Multimedia

Construction and Evaluation of Mandarin Multimodal Emotional Speech Database

Zhu Ting, Li Liangqi, Duan Shufei, Zhang Xueying, Xiao Zhongzhe, Jia Hairng, Liang Huizhi

专题命中 音频语音多模态 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.AI、eess.AS

FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wild

Zhi-Song Liu, Robin Courant, Vicky Kalogeiton

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 22 pages, 14 figures

AVSegFormer: Audio-Visual Segmentation with Transformer

Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, Tong Lu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、eess.AS

Comments 7 pages, 6 figures

Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations

Tao Meng, Yuntao Shou, Wei Ai, Nan Yin, Keqin Li

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Comments 16 pages, 9 figures

Sound of Story: Multi-modal Storytelling with Audio

Jaeyeon Bae, Seokhoon Jeong, Seokun Kang, Namgi Han, Jae-Yon Lee, Hyounghun Kim, Taehwan Kim

专题命中 音频语音多模态 ：multi-modal(title,abstract)；image-text(abstract)；分类 cs.MM、eess.AS

Comments Findings of EMNLP 2023, project: https://github.com/Sosdatasets/SoS_Dataset/

AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis

Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, Chenliang Xu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、eess.AS

Comments NeurIPS 2023

MIS-AVoiDD: Modality Invariant and Specific Representation for Audio-Visual Deepfake Detection

Vinaya Sree Katamneni, Ajita Rattani

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments 8 pages, 3 figures

Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser

Yung-Hsuan Lai, Yen-Chun Chen, Yu-Chiang Frank Wang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、eess.AS

Comments NeurIPS 2023

Class-Incremental Grouping Network for Continual Audio-Visual Learning

Shentong Mo, Weiguo Pian, Yapeng Tian

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments ICCV 2023. arXiv admin note: text overlap with arXiv:2303.17056

XFlow: Cross-modal Deep Neural Networks for Audiovisual Classification

Cătălina Cangea, Petar Veličković, Pietro Liò

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments Accepted at the IEEE ICDL-EPIROB 2017 Workshop on Computational Models for Crossmodal Learning (CMCML), 4 pages, 6 figures

Progressive Spatio-temporal Perception for Audio-Visual Question Answering

Guangyao Li, Wenxuan Hou, Di Hu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted by ACM MM 2023

Audio-Visual Deception Detection: DOLOS Dataset and Parameter-Efficient Crossmodal Learning

Xiaobao Guo, Nithish Muthuchamy Selvaraj, Zitong Yu, Adams Wai-Kin Kong, Bingquan Shen, Alex Kot

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments 11 pages, 6 figures

Perceptual Quality Assessment of Omnidirectional Audio-visual Signals

Xilei Zhu, Huiyu Duan, Yuqin Cao, Yuxin Zhu, Yucheng Zhu, Jing Liu, Li Chen, Xiongkuo Min, Guangtao Zhai

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、eess.AS

Comments 12 pages, 5 figures, to be published in CICAI2023

BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs

Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, Bingyi Kang

专题命中 音频语音多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

On Uni-Modal Feature Learning in Supervised Multi-Modal Learning

Chenzhuang Du, Jiaye Teng, Tingle Li, Yichen Liu, Tianyuan Yuan, Yue Wang, Yang Yuan, Hang Zhao

专题命中 音频语音多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

MMER: Multimodal Multi-task Learning for Speech Emotion Recognition

Sreyan Ghosh, Utkarsh Tyagi, S Ramaneswaran, Harshvardhan Srivastava, Dinesh Manocha

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Comments InterSpeech 2023 Main Conference

AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation

Rongjie Huang, Huadai Liu, Xize Cheng, Yi Ren, Linjun Li, Zhenhui Ye, Jinzheng He, Lichao Zhang, Jinglin Liu, Xiang Yin, Zhou Zhao

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Comments Accepted to ACL 2023

More Perspectives Mean Better: Underwater Target Recognition and Localization with Multimodal Data via Symbiotic Transformer and Multiview Regression

Shipei Liu, Xiaoya Fan, Guowei Wu

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.MM、eess.AS

Audio Retrieval for Multimodal Design Documents: A New Dataset and Algorithms

Prachi Singh, Srikrishna Karanam, Sumit Shekhar

专题命中 音频语音多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.MM、eess.AS

Comments 5 pages including references

Canonical Cortical Graph Neural Networks and its Application for Speech Enhancement in Audio-Visual Hearing Aids

Leandro A. Passos, João Paulo Papa, Amir Hussain, Ahsan Adeel

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、eess.AS

Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection

Xuanjun Chen, Haibin Wu, Helen Meng, Hung-yi Lee, Jyh-Shing Roger Jang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.MM、eess.AS

Comments Accepted by SLT 2022

Predict-and-Update Network: Audio-Visual Speech Recognition Inspired by Human Speech Perception

Jiadong Wang, Xinyuan Qian, Haizhou Li

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.MM、eess.AS

AudioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separation

Efthymios Tzinis, Scott Wisdom, Tal Remez, John R. Hershey

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、eess.AS

Comments ECCV 2022

Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detection

Jiashuo Yu, Jinyu Liu, Ying Cheng, Rui Feng, Yuejie Zhang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM

Comments ACM MM 2022

VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voices

Venkatesh S. Kadandale, Juan F. Montesinos, Gloria Haro

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、eess.AS

Comments Paper accepted to Interspeech 2022; Project Page: https://ipcv.github.io/VocaLiST/