多模态大模型 - arXivDaily 专题

2412.16861 2024-12-31 cs.SD cs.CV cs.MM eess.AS 85%

SoundLoc3D: Invisible 3D Sound Source Localization and Classification Using a Multimodal RGB-D Acoustic Camera

Yuhang He, Sangyun Shin, Anoop Cherian, Niki Trigoni, Andrew Markham

专题命中音频语音多模态：multimodal(title)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by WACV2025

Journal ref WACV2025

2412.13543 2024-12-19 cs.CV cs.AI cs.CL 85%

Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning

Yunbin Tu, Liang Li, Li Su, Qingming Huang

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by AAAI 2025

2412.09596 2024-12-13 cs.CV cs.AI cs.CL 85%

InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions

Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, Qipeng Guo, Haodong Duan, Xin Chen, Han Lv, Zheng Nie, Min Zhang, Bin Wang, Wenwei Zhang, Xinyue Zhang, Jiaye Ge, Wei Li, Jingwen Li, Zhongying Tu, Conghui He, Xingcheng Zhang, Kai Chen, Yu Qiao, Dahua Lin, Jiaqi Wang

专题命中音频语音多模态：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Github Repo: https://github.com/InternLM/InternLM-XComposer/tree/main/InternLM-XComposer-2.5-OmniLive

2412.09168 2024-12-13 cs.SD cs.CV cs.MM eess.AS 85%

YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controls

Zihao Chen, Haomin Zhang, Xinhan Di, Haoyu Wang, Sizhe Shan, Junjie Zheng, Yunming Liang, Yihan Fan, Xinfa Zhu, Wenjie Tian, Yihua Wang, Chaofan Ding, Lei Xie

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 16 pages, 4 figures

2412.07406 2024-12-11 cs.CV cs.MM cs.SD eess.AS 85%

Learning Self-Supervised Audio-Visual Representations for Sound Recommendations

Sudha Krishnamurthy

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Published in the Proceedings of the International Symposium on Visual Computing, 2021 https://dl.acm.org/doi/10.1007/978-3-030-90436-4_10

2412.06209 2024-12-10 cs.CV cs.MM cs.SD eess.AS 85%

Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignment

Kim Sung-Bin, Arda Senocak, Hyunwoo Ha, Tae-Hyun Oh

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Under-review

2411.02860 2024-11-06 cs.CV cs.LG cs.MM cs.SD eess.AS 85%

Continual Audio-Visual Sound Separation

Weiguo Pian, Yiyang Nan, Shijian Deng, Shentong Mo, Yunhui Guo, Yapeng Tian

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments NeurIPS 2024

2309.07115 2024-09-25 cs.SD cs.CV cs.LG cs.MM eess.AS 85%

Getting More for Less: Using Weak Labels and AV-Mixup for Robust Audio-Visual Speaker Verification

Anith Selvakumar, Homa Fashandi

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to INTERSPEECH 2024

Journal ref Proc. Interspeech 2024, 4728-4732

2408.00624 2024-08-02 eess.AS cs.CL cs.CV 85%

SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Data

Yichen Lu, Jiaqi Song, Xuankai Chang, Hengwei Bian, Soumi Maiti, Shinji Watanabe

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL、eess.AS

2407.16171 2024-07-25 cs.CV cs.AI cs.MM 85%

Learning Trimodal Relation for Audio-Visual Question Answering with Missing Modality

Kyu Ri Park, Hong Joo Lee, Jung Uk Kim

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted at ECCV 2024

2407.15046 2024-07-23 cs.CV cs.CL cs.MM 85%

Audio-visual training for improved grounding in video-text LLMs

Shivprasad Sagare, Hemachandran S, Kinshuk Sarabhai, Prashant Ullegaddi, Rajeshkumar SA

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.MM

2407.13676 2024-07-19 cs.MM cs.CV cs.SD eess.AS 85%

Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment

Arda Senocak, Hyeonggon Ryu, Junsik Kim, Tae-Hyun Oh, Hanspeter Pfister, Joon Son Chung

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Journal Extension of ICCV 2023 paper (arXiV:2309.10724). Code is available at https://github.com/kaistmm/SSLalignment

2407.13095 2024-07-19 cs.CV cs.LG cs.MM cs.SD eess.AS 85%

Audio-visual Generalized Zero-shot Learning the Easy Way

Shentong Mo, Pedro Morgado

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

2407.08126 2024-07-12 cs.AI cs.CV cs.MM 85%

Label-anticipated Event Disentanglement for Audio-Visual Video Parsing

Jinxing Zhou, Dan Guo, Yuxin Mao, Yiran Zhong, Xiaojun Chang, Meng Wang

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by ECCV2024

2407.01851 2024-07-04 cs.CV cs.AI cs.LG eess.AS 85%

Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time

Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, Dinesh Manocha

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Accepted at ECCV 2024

2401.09759 2024-07-03 cs.CV cs.MM cs.SD eess.AS 85%

SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognition

Hao Wang, Shuhei Kurita, Shuichiro Shimizu, Daisuke Kawahara

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 3rd Workshop on Advances in Language and Vision Research (ALVR 2024)

2406.02951 2024-06-06 cs.CV cs.MM cs.SD eess.AS 85%

AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection

Trevine Oorloff, Surya Koppisetti, Nicolò Bonettini, Divyaraj Solanki, Ben Colman, Yaser Yacoob, Ali Shahriyari, Gaurav Bharaj

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to CVPR 2024

2405.14598 2024-05-27 cs.CV cs.LG cs.MM cs.SD eess.AS 85%

Visual Echoes: A Simple Unified Transformer for Audio-Visual Generation

Shiqi Yang, Zhi Zhong, Mengjie Zhao, Shusuke Takahashi, Masato Ishii, Takashi Shibuya, Yuki Mitsufuji

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 10 pages

2404.13789 2024-04-24 cs.SD cs.AI cs.IR cs.MM eess.AS 85%

Anchor-aware Deep Metric Learning for Audio-visual Retrieval

Donghuo Zeng, Yanan Wang, Kazushi Ikeda, Yi Yu

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 9 pages, 5 figures. Accepted by ACM ICMR 2024

2302.10035 2024-04-11 cs.CV cs.AI cs.MM 85%

Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey

Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao-Yong Wei, Yaowei Wang, Yonghong Tian, Wen Gao

专题命中音频语音多模态：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by Machine Intelligence Research (MIR)

2403.04245 2024-03-08 cs.SD cs.CV cs.LG cs.MM eess.AS 85%

A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition

Yusheng Dai, Hang Chen, Jun Du, Ruoyu Wang, Shihao Chen, Jiefeng Ma, Haotian Wang, Chin-Hui Lee

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments the paper is accepted by CVPR2024

2312.01017 2023-12-05 cs.CV cs.AI cs.LG cs.MM cs.SD 85%

Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling

Shentong Mo, Pedro Morgado

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

2310.20446 2023-11-01 cs.SD cs.CV cs.MM eess.AS 85%

LAVSS: Location-Guided Audio-Visual Spatial Audio Separation

Yuxin Ye, Wenming Yang, Yapeng Tian

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by WACV2024

2310.13451 2023-10-23 cs.SD cs.CV cs.IR cs.MM eess.AS 85%

Two-Stage Triplet Loss Training with Curriculum Augmentation for Audio-Visual Retrieval

Donghuo Zeng, Kazushi Ikeda

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 8 pages, 6 figures

2309.16569 2023-09-29 cs.SD cs.CV cs.MM eess.AS 85%

Audio-Visual Speaker Verification via Joint Cross-Attention

R. Gnana Praveen, Jahangir Alam

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

2309.06728 2023-09-14 cs.CV cs.MM cs.SD eess.AS 85%

Leveraging Foundation models for Unsupervised Audio-Visual Segmentation

Swapnil Bhosale, Haosen Yang, Diptesh Kanojia, Xiatian Zhu

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

2308.13421 2023-08-28 cs.CV cs.MM cs.SD eess.AS 85%

Exploiting Diverse Feature for Multimodal Sentiment Analysis

Jia Li, Wei Qian, Kun Li, Qi Li, Dan Guo, Meng Wang

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

2308.09322 2023-08-21 cs.CV cs.AI cs.MM 85%

Audio-Visual Glance Network for Efficient Video Recognition

Muhammad Adi Nugroho, Sangmin Woo, Sumin Lee, Changick Kim

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments ICCV 2023

2307.15097 2023-07-31 cs.CL cs.LG cs.MM eess.AS 85%

Cascaded Cross-Modal Transformer for Request and Complaint Detection

Nicolae-Catalin Ristea, Radu Tudor Ionescu

专题命中音频语音多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted at ACMMM 2023

2307.13236 2023-07-26 cs.SD cs.CV cs.LG cs.MM eess.AS 85%

Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation

Jinxiang Liu, Chen Ju, Chaofan Ma, Yanfeng Wang, Yu Wang, Ya Zhang

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments arXiv admin note: text overlap with arXiv:2305.11019

1. 音频语音多模态 4567 篇

SoundLoc3D: Invisible 3D Sound Source Localization and Classification Using a Multimodal RGB-D Acoustic Camera

Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning

InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions

YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controls

Learning Self-Supervised Audio-Visual Representations for Sound Recommendations

Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignment

Continual Audio-Visual Sound Separation

Getting More for Less: Using Weak Labels and AV-Mixup for Robust Audio-Visual Speaker Verification

SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Data

Learning Trimodal Relation for Audio-Visual Question Answering with Missing Modality

Audio-visual training for improved grounding in video-text LLMs

Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment

Audio-visual Generalized Zero-shot Learning the Easy Way

Label-anticipated Event Disentanglement for Audio-Visual Video Parsing

Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time

SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognition

AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection

Visual Echoes: A Simple Unified Transformer for Audio-Visual Generation

Anchor-aware Deep Metric Learning for Audio-visual Retrieval

Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey

A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition

Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling

LAVSS: Location-Guided Audio-Visual Spatial Audio Separation

Two-Stage Triplet Loss Training with Curriculum Augmentation for Audio-Visual Retrieval

Audio-Visual Speaker Verification via Joint Cross-Attention

Leveraging Foundation models for Unsupervised Audio-Visual Segmentation

Exploiting Diverse Feature for Multimodal Sentiment Analysis

Audio-Visual Glance Network for Efficient Video Recognition

Cascaded Cross-Modal Transformer for Request and Complaint Detection

Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation