多模态大模型

SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Data

Yichen Lu, Jiaqi Song, Xuankai Chang, Hengwei Bian, Soumi Maiti, Shinji Watanabe

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL、eess.AS

Learning Trimodal Relation for Audio-Visual Question Answering with Missing Modality

Kyu Ri Park, Hong Joo Lee, Jung Uk Kim

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted at ECCV 2024

Audio-visual training for improved grounding in video-text LLMs

Shivprasad Sagare, Hemachandran S, Kinshuk Sarabhai, Prashant Ullegaddi, Rajeshkumar SA

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.MM

Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment

Arda Senocak, Hyeonggon Ryu, Junsik Kim, Tae-Hyun Oh, Hanspeter Pfister, Joon Son Chung

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Journal Extension of ICCV 2023 paper (arXiV:2309.10724). Code is available at https://github.com/kaistmm/SSLalignment

Audio-visual Generalized Zero-shot Learning the Easy Way

Shentong Mo, Pedro Morgado

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Label-anticipated Event Disentanglement for Audio-Visual Video Parsing

Jinxing Zhou, Dan Guo, Yuxin Mao, Yiran Zhong, Xiaojun Chang, Meng Wang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by ECCV2024

Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time

Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, Dinesh Manocha

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Accepted at ECCV 2024

SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognition

Hao Wang, Shuhei Kurita, Shuichiro Shimizu, Daisuke Kawahara

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 3rd Workshop on Advances in Language and Vision Research (ALVR 2024)

AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection

Trevine Oorloff, Surya Koppisetti, Nicolò Bonettini, Divyaraj Solanki, Ben Colman, Yaser Yacoob, Ali Shahriyari, Gaurav Bharaj

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to CVPR 2024

Visual Echoes: A Simple Unified Transformer for Audio-Visual Generation

Shiqi Yang, Zhi Zhong, Mengjie Zhao, Shusuke Takahashi, Masato Ishii, Takashi Shibuya, Yuki Mitsufuji

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 10 pages

Anchor-aware Deep Metric Learning for Audio-visual Retrieval

Donghuo Zeng, Yanan Wang, Kazushi Ikeda, Yi Yu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 9 pages, 5 figures. Accepted by ACM ICMR 2024

Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey

Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao-Yong Wei, Yaowei Wang, Yonghong Tian, Wen Gao

专题命中 音频语音多模态 ：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by Machine Intelligence Research (MIR)

A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition

Yusheng Dai, Hang Chen, Jun Du, Ruoyu Wang, Shihao Chen, Jiefeng Ma, Haotian Wang, Chin-Hui Lee

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments the paper is accepted by CVPR2024

Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling

Shentong Mo, Pedro Morgado

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

LAVSS: Location-Guided Audio-Visual Spatial Audio Separation

Yuxin Ye, Wenming Yang, Yapeng Tian

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by WACV2024

Two-Stage Triplet Loss Training with Curriculum Augmentation for Audio-Visual Retrieval

Donghuo Zeng, Kazushi Ikeda

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 8 pages, 6 figures

Audio-Visual Speaker Verification via Joint Cross-Attention

R. Gnana Praveen, Jahangir Alam

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Leveraging Foundation models for Unsupervised Audio-Visual Segmentation

Swapnil Bhosale, Haosen Yang, Diptesh Kanojia, Xiatian Zhu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Exploiting Diverse Feature for Multimodal Sentiment Analysis

Jia Li, Wei Qian, Kun Li, Qi Li, Dan Guo, Meng Wang

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Audio-Visual Glance Network for Efficient Video Recognition

Muhammad Adi Nugroho, Sangmin Woo, Sumin Lee, Changick Kim

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments ICCV 2023

Cascaded Cross-Modal Transformer for Request and Complaint Detection

Nicolae-Catalin Ristea, Radu Tudor Ionescu

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted at ACMMM 2023

Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation

Jinxiang Liu, Chen Ju, Chaofan Ma, Yanfeng Wang, Yu Wang, Ya Zhang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments arXiv admin note: text overlap with arXiv:2305.11019

Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding

Zengjie Song, Zhaoxiang Zhang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to IEEE Transactions on Neural Networks and Learning Systems (T-NNLS)

MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition

Yuchen Hu, Chen Chen, Ruizhe Li, Heqing Zou, Eng Siong Chng

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 14 pages, 5 figures, Accepted by ACL 2023

Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention

Xubo Liu, Qiushi Huang, Xinhao Mei, Haohe Liu, Qiuqiang Kong, Jianyuan Sun, Shengchen Li, Tom Ko, Yu Zhang, Lilian H. Tang, Mark D. Plumbley, Volkan Kılıç, Wenwu Wang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments INTERSPEECH 2023

AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation

Shentong Mo, Yapeng Tian

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline

Tiantian Geng, Teng Wang, Jinming Duan, Runmin Cong, Feng Zheng

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by CVPR2023

Egocentric Audio-Visual Object Localization

Chao Huang, Yapeng Tian, Anurag Kumar, Chenliang Xu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by CVPR 2023

Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring

Joanna Hong, Minsu Kim, Jeongsoo Choi, Yong Man Ro

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted at CVPR 2023. Implementation available: https://github.com/joannahong/AV-RelScore

Speaker Recognition in Realistic Scenario Using Multimodal Data

Saqlain Hussain Shah, Muhammad Saad Saeed, Shah Nawaz, Muhammad Haroon Yousaf

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted at the International Conference on Artificial Intelligence (ICAI'2023)