多模态大模型 - arXivDaily 专题

2405.10690 2024-07-16 cs.CV 83%

CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsing

Faegheh Sardari, Armin Mustafa, Philip J. B. Jackson, Adrian Hilton

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted at ECCV 2024

2407.01034 2024-07-02 cs.CV cs.GR 83%

Enhancing Speech-Driven 3D Facial Animation with Audio-Visual Guidance from Lip Reading Expert

Han EunGi, Oh Hyun-Bin, Kim Sung-Bin, Corentin Nivelet Etcheberry, Suekyeong Nam, Janghoon Joo, Tae-Hyun Oh

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments INTERSPEECH 2024

2406.17605 2024-06-26 cs.MM cs.AI cs.CL cs.CV cs.IR 83%

NativE: Multi-modal Knowledge Graph Completion in the Wild

Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Binbin Hu, Ziqi Liu, Wen Zhang, Huajun Chen

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by SIGIR 2024 as a full paper

2406.13894 2024-06-21 cs.CV cs.CY 83%

Using Multimodal Large Language Models for Automated Detection of Traffic Safety Critical Events

Mohammad Abu Tami, Huthaifa I. Ashqar, Mohammed Elhenawy

专题命中音频语音多模态：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

2406.10448 2024-06-18 eess.AS cs.SD 83%

AVR: Synergizing Foundation Models for Audio-Visual Humor Detection

Sarthak Sharma, Orchid Chetia Phukan, Drishti Singh, Arun Balaji Buduru, Rajesh Sharma

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 eess.AS

Comments Accepted to INTERSPEECH 2024 Show & Tell Demonstrations

2405.20773 2024-06-13 cs.CR cs.AI 83%

Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Character

Siyuan Ma, Weidi Luo, Yu Wang, Xiaogeng Liu

专题命中音频语音多模态：multimodal(title,abstract)；MLLM(abstract)；分类 cs.AI

2406.06163 2024-06-11 cs.CV 83%

Extending Segment Anything Model into Auditory and Temporal Dimensions for Audio-Visual Segmentation

Juhyeong Seon, Woobin Im, Sebin Lee, Jumin Lee, Sung-Eui Yoon

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted to ICIP 2024

2405.12983 2024-05-24 eess.AS cs.AI cs.CV cs.MM cs.SD 83%

Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer

Maxime Burchi, Krishna C. Puvvada, Jagadeesh Balam, Boris Ginsburg, Radu Timofte

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

2311.12268 2024-04-25 cs.CV 83%

Boosting Audio-visual Zero-shot Learning with Large Language Models

Haoxing Chen, Yaohui Li, Yan Hong, Zizheng Huang, Zhuoer Xu, Zhangxuan Gu, Jun Lan, Huijia Zhu, Weiqiang Wang

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

2312.12870 2024-04-04 cs.CV 83%

The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective

Wenqi Jia, Miao Liu, Hao Jiang, Ishwarya Ananthabhotla, James M. Rehg, Vamsi Krishna Ithapu, Ruohan Gao

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

2404.00861 2024-04-02 eess.AS eess.IV 83%

Enhancing Real-World Active Speaker Detection with Multi-Modal Extraction Pre-Training

Ruijie Tao, Xinyuan Qian, Rohan Kumar Das, Xiaoxue Gao, Jiadong Wang, Haizhou Li

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 eess.AS

Comments 10 pages

2403.02969 2024-03-26 cs.CV 83%

Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception

Junwen He, Yifan Wang, Lijun Wang, Huchuan Lu, Jun-Yan He, Jin-Peng Lan, Bin Luo, Xuansong Xie

专题命中音频语音多模态：multi-modal(title)；multimodal(abstract)；MLLM(abstract)；分类 cs.CV

Comments CVPR 2024

2403.11074 2024-03-19 cs.CV cs.AI cs.MM cs.SD eess.AS 83%

Audio-Visual Segmentation via Unlabeled Frame Exploitation

Jinxiang Liu, Yikun Liu, Fei Zhang, Chen Ju, Ya Zhang, Yanfeng Wang

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by CVPR 2024

2402.16757 2024-02-27 cs.SD eess.AS 83%

Towards Environmental Preference Based Speech Enhancement For Individualised Multi-Modal Hearing Aids

Jasper Kirton-Wingate, Shafique Ahmed, Adeel Hussain, Mandar Gogate, Kia Dashtipour, Jen-Cheng Hou, Tassadaq Hussain, Yu Tsao, Amir Hussain

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 eess.AS

Comments This has been submitted to the Trends in Hearing journal

2402.16394 2024-02-27 eess.AS cs.SD 83%

Audio-Visual Speech Enhancement in Noisy Environments via Emotion-Based Contextual Cues

Tassadaq Hussain, Kia Dashtipour, Yu Tsao, Amir Hussain

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 eess.AS

2305.18797 2024-02-14 cs.CV 83%

Learning Weakly Supervised Audio-Visual Violence Detection in Hyperbolic Space

Xiaogang Peng, Hao Wen, Yikai Luo, Xiao Zhou, Keyang Yu, Ping Yang, Zizhao Wu

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments 11 pages, 12 figures, typos are fixed

2401.09150 2024-01-18 cs.CL 83%

Bridging Research and Readers: A Multi-Modal Automated Academic Papers Interpretation System

Feng Jiang, Kuang Wang, Haizhou Li

专题命中音频语音多模态：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CL

2401.05746 2024-01-12 cs.MM 83%

Cross-Modality and Within-Modality Regularization for Audio-Visual DeepFake Detection

Heqing Zou, Meng Shen, Yuchen Hu, Chen Chen, Eng Siong Chng, Deepu Rajan

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.MM

Comments Accepted by ICASSP 2024

2401.04023 2024-01-09 cs.CV cs.AI cs.LG cs.MM cs.SD eess.AS 83%

Efficient Multiscale Multimodal Bottleneck Transformer for Audio-Video Classification

Wentao Zhu

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by WACV 2024; well-formatted PDF is in https://drive.google.com/file/d/10Zo_ydJZFAm7YsxHDgTjhyc4dEJbW_dk/view?usp=sharing

2401.03468 2024-01-09 eess.AS cs.SD 83%

Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation

Qiushi Zhu, Jie Zhang, Yu Gu, Yuchen Hu, Lirong Dai

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 eess.AS

Comments Accepted by AAAI 2024

2309.05396 2023-12-27 cs.SD eess.AS 83%

SlideSpeech: A Large-Scale Slide-Enriched Audio-Visual Corpus

Haoxu Wang, Fan Yu, Xian Shi, Yuezhang Wang, Shiliang Zhang, Ming Li

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 eess.AS

Comments Accepted by ICASSP 2024

2312.12816 2023-12-21 cs.CV 83%

Object-aware Adaptive-Positivity Learning for Audio-Visual Question Answering

Zhangbin Li, Dan Guo, Jinxing Zhou, Jing Zhang, Meng Wang

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted by AAAI-2024

2312.06720 2023-12-15 cs.CV 83%

Audio-Visual LLM for Video Understanding

Fangxun Shu, Lei Zhang, Hao Jiang, Cihang Xie

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

2311.17655 2023-11-30 cs.CV cs.AI cs.MM cs.SD eess.AS 83%

Vulnerability of Automatic Identity Recognition to Audio-Visual Deepfakes

Pavel Korshunov, Haolin Chen, Philip N. Garner, Sebastien Marcel

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 10 pages, 3 figures, 3 tables

2311.15080 2023-11-28 cs.CV cs.AI cs.LG cs.MM cs.SD eess.AS 83%

Weakly-Supervised Audio-Visual Segmentation

Shentong Mo, Bhiksha Raj

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

2311.04766 2023-11-14 cs.CV 83%

DualTalker: A Cross-Modal Dual Learning Approach for Speech-Driven 3D Facial Animation

Guinan Su, Yanwu Yang, Zhifeng Li

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV

2310.16477 2023-10-26 cs.CV 83%

Show from Tell: Audio-Visual Modelling in Clinical Settings

Jianbo Jiao, Mohammad Alsharid, Lior Drukker, Aris T. Papageorghiou, Andrew Zisserman, J. Alison Noble

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

2310.06383 2023-10-11 cs.AI 83%

What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?

Siting Li, Chenzhuang Du, Yue Zhao, Yu Huang, Hang Zhao

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.AI

2310.05193 2023-10-10 cs.CV 83%

Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models

Chenzhuang Du, Yue Zhao, Chonghua Liao, Jiacheng You, Jie Fu, Hang Zhao

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV

2309.17395 2023-10-02 cs.LG cs.SD eess.AS stat.ML 83%

AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognition

Andrew Rouditchenko, Ronan Collobert, Tatiana Likhomanenko

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 eess.AS

Comments Under review

1. 音频语音多模态 4567 篇

CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsing

Enhancing Speech-Driven 3D Facial Animation with Audio-Visual Guidance from Lip Reading Expert

NativE: Multi-modal Knowledge Graph Completion in the Wild

Using Multimodal Large Language Models for Automated Detection of Traffic Safety Critical Events

AVR: Synergizing Foundation Models for Audio-Visual Humor Detection

Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Character

Extending Segment Anything Model into Auditory and Temporal Dimensions for Audio-Visual Segmentation

Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer

Boosting Audio-visual Zero-shot Learning with Large Language Models

The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective

Enhancing Real-World Active Speaker Detection with Multi-Modal Extraction Pre-Training

Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception

Audio-Visual Segmentation via Unlabeled Frame Exploitation

Towards Environmental Preference Based Speech Enhancement For Individualised Multi-Modal Hearing Aids

Audio-Visual Speech Enhancement in Noisy Environments via Emotion-Based Contextual Cues

Learning Weakly Supervised Audio-Visual Violence Detection in Hyperbolic Space

Bridging Research and Readers: A Multi-Modal Automated Academic Papers Interpretation System

Cross-Modality and Within-Modality Regularization for Audio-Visual DeepFake Detection

Efficient Multiscale Multimodal Bottleneck Transformer for Audio-Video Classification

Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation

SlideSpeech: A Large-Scale Slide-Enriched Audio-Visual Corpus

Object-aware Adaptive-Positivity Learning for Audio-Visual Question Answering

Audio-Visual LLM for Video Understanding

Vulnerability of Automatic Identity Recognition to Audio-Visual Deepfakes

Weakly-Supervised Audio-Visual Segmentation

DualTalker: A Cross-Modal Dual Learning Approach for Speech-Driven 3D Facial Animation

Show from Tell: Audio-Visual Modelling in Clinical Settings

What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?

Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models

AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognition