多模态大模型 - arXivDaily 专题

2307.07859 2023-07-20 cs.CV 83%

Unified Adversarial Patch for Cross-modal Attacks in the Physical World

Xingxing Wei, Yao Huang, Yitong Sun, Jie Yu

专题命中音频语音多模态：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments 10 pages, 8 figures, accepted by ICCV2023

2305.01979 2023-07-18 cs.CV 83%

Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localization

Zhixi Cai, Shreya Ghosh, Abhinav Dhall, Tom Gedeon, Kalin Stefanov, Munawar Hayat

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments The paper is under consideration/review at Computer Vision and Image Understanding Journal

2307.07277 2023-07-17 cs.CL 83%

Are words equally surprising in audio and audio-visual comprehension?

Pranava Madhyastha, Ye Zhang, Gabriella Vigliocco

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CL

Comments In CogSci 2023

2306.16034 2023-06-29 cs.AI cs.NI 83%

Stone Needle: A General Multimodal Large-scale Model Framework towards Healthcare

Weihua Liu, Yong Zuo

专题命中音频语音多模态：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.AI

2306.15644 2023-06-28 cs.CL 83%

Style-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos

Chiori Hori, Puyuan Peng, David Harwath, Xinyu Liu, Kei Ota, Siddarth Jain, Radu Corcodel, Devesh Jha, Diego Romeres, Jonathan Le Roux

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CL

Comments Accepted to Interspeech2023

2306.09431 2023-06-19 cs.MM 83%

Towards Long Form Audio-visual Video Understanding

Wenxuan Hou, Guangyao Li, Yapeng Tian, Di Hu

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.MM

2306.02263 2023-06-06 cs.SD cs.CV 83%

MAVD: The First Open Large-Scale Mandarin Audio-Visual Dataset with Depth Information

Jianrong Wang, Yuchen Huo, Li Liu, Tianyi Xu, Qi Li, Sen Li

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

2305.11000 2023-05-22 cs.CL 83%

SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities

Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, Xipeng Qiu

专题命中音频语音多模态：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CL

Comments work in progress

2304.12082 2023-04-25 cs.SD eess.AS 83%

Deep Audio-Visual Singing Voice Transcription based on Self-Supervised Learning Models

Xiangming Gu, Wei Zeng, Jianan Zhang, Longshen Ou, Ye Wang

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 eess.AS

2303.14768 2023-03-28 cs.CV 83%

Collaborative Noisy Label Cleaner: Learning Scene-aware Trailers for Multi-modal Highlight Detection in Movies

Bei Gan, Xiujun Shu, Ruizhi Qiao, Haoqian Wu, Keyu Chen, Hanjun Li, Bo Ren

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV

Comments Accepted to CVPR2023

2204.13386 2023-03-21 cs.CV 83%

Self-supervised Contrastive Learning for Audio-Visual Action Recognition

Yang Liu, Ying Tan, Haoyuan Lan

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments 6 pages, 2 figures

2303.06341 2023-03-14 eess.AS 83%

The NPU-ASLP System for Audio-Visual Speech Recognition in MISP 2022 Challenge

Pengcheng Guo, He Wang, Bingshen Mu, Ao Zhang, Peikun Chen

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 eess.AS

Comments 2 pages, accepted by ICASSP 2023

2303.06129 2023-03-13 cs.CV 83%

Single-branch Network for Multimodal Training

Muhammad Saad Saeed, Shah Nawaz, Muhammad Haris Khan, Muhammad Zaigham Zaheer, Karthik Nandakumar, Muhammad Haroon Yousaf, Arif Mahmood

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted at ICASSP 2023

2301.09595 2023-02-23 cs.CV 83%

Zorro: the masked multimodal transformer

Adrià Recasens, Jason Lin, Joāo Carreira, Drew Jaegle, Luyu Wang, Jean-baptiste Alayrac, Pauline Luc, Antoine Miech, Lucas Smaira, Ross Hemsley, Andrew Zisserman

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV

2302.07702 2023-02-16 cs.CV 83%

Audio-Visual Contrastive Learning with Temporal Self-Supervision

Simon Jenni, Alexander Black, John Collomosse

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments AAAI-23

2302.05727 2023-02-14 cs.CV 83%

Flexible-modal Deception Detection with Audio-Visual Adapter

Zhaoxu Li, Zitong Yu, Nithish Muthuchamy Selvaraj, Xiaobao Guo, Bingquan Shen, Adams Wai-Kin Kong, Alex Kot

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

2302.02447 2023-02-07 eess.AS cs.SD 83%

cross-modal fusion techniques for utterance-level emotion recognition from text and speech

Jiachen Luo, Huy Phan, Joshua Reiss

专题命中音频语音多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 eess.AS

Comments 6 pages, 2 figures

2107.07268 2022-12-13 cs.MM cs.IR 83%

Cross-modal Variational Auto-encoder for Content-based Micro-video Background Music Recommendation

Jing Yi, Yaochen Zhu, Jiayi Xie, Zhenzhong Chen

专题命中音频语音多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.MM

2211.10883 2022-11-22 cs.CV 83%

Audio-visual video face hallucination with frequency supervision and cross modality support by speech based lip reading loss

Shailza Sharma, Abhinav Dhall, Vinay Kumar, Vivek Singh Bawa

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV

2205.12216 2022-11-11 cs.CL 83%

T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation

Paul-Ambroise Duquenne, Hongyu Gong, Benoît Sagot, Holger Schwenk

专题命中音频语音多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CL

2210.15903 2022-10-31 eess.AS cs.SD eess.SP 83%

Speaker recognition with two-step multi-modal deep cleansing

Ruijie Tao, Kong Aik Lee, Zhan Shi, Haizhou Li

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 eess.AS

Comments 5 pages, 3 figures

2210.14581 2022-10-27 eess.AS cs.SD eess.IV 83%

Deep Learning Based Audio-Visual Multi-Speaker DOA Estimation Using Permutation-Free Loss Function

Qing Wang, Hang Chen, Ya Jiang, Zhe Wang, Yuyang Wang, Jun Du, Chin-Hui Lee

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 eess.AS

Comments 5 pages, 3 figures, accepted by ISCSLP 2022

2209.09068 2022-09-20 cs.CV 83%

Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention

R Gnana Praveen, Eric Granger, Patrick Cardinal

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments arXiv admin note: substantial text overlap with arXiv:2203.14779, arXiv:2111.05222

2209.08000 2022-09-19 cs.MM 83%

TIMIT-TTS: a Text-to-Speech Dataset for Multimodal Synthetic Media Detection

Davide Salvi, Brian Hosler, Paolo Bestagini, Matthew C. Stamm, Stefano Tubaro

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.MM

2208.05220 2022-08-17 cs.CV 83%

Dual Domain-Adversarial Learning for Audio-Visual Saliency Prediction

Yingzi Fan, Longfei Han, Yue Zhang, Lechao Cheng, Chen Xia, Di Hu

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by ACM MM workshop 2022(HCMA2022)

2207.06020 2022-07-14 cs.SD cs.AI cs.CV cs.MM eess.AS eess.IV 83%

Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition

Joanna Hong, Minsu Kim, Daehun Yoo, Yong Man Ro

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted at Interspeech 2022

2202.07428 2022-07-12 eess.IV cs.SD eess.AS 83%

Learning Contextually Fused Audio-visual Representations for Audio-visual Speech Recognition

Zi-Qiang Zhang, Jie Zhang, Jian-Shu Zhang, Ming-Hui Wu, Xin Fang, Li-Rong Dai

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 eess.AS

Comments 5 pages

2203.14072 2022-04-06 cs.CV 83%

Learning to Answer Questions in Dynamic Audio-Visual Scenarios

Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, Di Hu

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Accepted by CVPR2022 (Oral presentation)

2202.09263 2022-02-21 cs.LG cs.MM 83%

Is Cross-Attention Preferable to Self-Attention for Multi-Modal Emotion Recognition?

Vandana Rajan, Alessio Brutti, Andrea Cavallaro

专题命中音频语音多模态：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.MM

Comments Accepted at ICASSP 2022

2202.09081 2022-02-21 eess.AS cs.AI cs.CV cs.MM cs.SD eess.IV 83%

VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversion

Disong Wang, Shan Yang, Dan Su, Xunying Liu, Dong Yu, Helen Meng

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted to ICASSP 2022. Demo page is available at https://wendison.github.io/VCVTS-demo/

1. 音频语音多模态 4578 篇

Unified Adversarial Patch for Cross-modal Attacks in the Physical World

Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localization

Are words equally surprising in audio and audio-visual comprehension?

Stone Needle: A General Multimodal Large-scale Model Framework towards Healthcare

Style-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos

Towards Long Form Audio-visual Video Understanding

MAVD: The First Open Large-Scale Mandarin Audio-Visual Dataset with Depth Information

SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities

Deep Audio-Visual Singing Voice Transcription based on Self-Supervised Learning Models

Collaborative Noisy Label Cleaner: Learning Scene-aware Trailers for Multi-modal Highlight Detection in Movies

Self-supervised Contrastive Learning for Audio-Visual Action Recognition

The NPU-ASLP System for Audio-Visual Speech Recognition in MISP 2022 Challenge

Single-branch Network for Multimodal Training

Zorro: the masked multimodal transformer

Audio-Visual Contrastive Learning with Temporal Self-Supervision

Flexible-modal Deception Detection with Audio-Visual Adapter

cross-modal fusion techniques for utterance-level emotion recognition from text and speech

Cross-modal Variational Auto-encoder for Content-based Micro-video Background Music Recommendation

Audio-visual video face hallucination with frequency supervision and cross modality support by speech based lip reading loss

T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation

Speaker recognition with two-step multi-modal deep cleansing

Deep Learning Based Audio-Visual Multi-Speaker DOA Estimation Using Permutation-Free Loss Function

Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention

TIMIT-TTS: a Text-to-Speech Dataset for Multimodal Synthetic Media Detection

Dual Domain-Adversarial Learning for Audio-Visual Saliency Prediction

Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition

Learning Contextually Fused Audio-visual Representations for Audio-visual Speech Recognition

Learning to Answer Questions in Dynamic Audio-Visual Scenarios

Is Cross-Attention Preferable to Self-Attention for Multi-Modal Emotion Recognition?

VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversion