多模态大模型 - arXivDaily 专题

2210.07839 2023-04-13 cs.MM cs.CV cs.SD eess.AS 82%

Contrastive Audio-Visual Masked Autoencoder

Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, James Glass

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted at ICLR 2023 as a notable top 25% paper. Code and pretrained models are at https://github.com/yuangongnd/cav-mae

2212.07983 2023-04-06 cs.CV cs.CL cs.LG cs.SD eess.AS 82%

Vision Transformers are Parameter-Efficient Audio-Visual Learners

Yan-Bo Lin, Yi-Lin Sung, Jie Lei, Mohit Bansal, Gedas Bertasius

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments CVPR 2023 Project Page: https://genjib.github.io/project_page/LAVISH/

2303.06904 2023-03-14 cs.CV cs.AI cs.CL 82%

Contextually-rich human affect perception using multimodal scene information

Digbalay Bose, Rajat Hebbar, Krishna Somandepalli, Shrikanth Narayanan

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023

2207.05042 2023-02-20 cs.CV cs.MM cs.SD eess.AS eess.IV 82%

Audio-Visual Segmentation

Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, Yiran Zhong

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments ECCV 2022; Code is available at https://github.com/OpenNLPLab/AVSBench

2212.05301 2023-02-03 cs.SD cs.CV cs.MM eess.AS 82%

Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning

Chen Chen, Yuchen Hu, Qiang Zhang, Heqing Zou, Beier Zhu, Eng Siong Chng

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by AAAI2023

2301.01456 2023-01-05 cs.CV cs.CL cs.SD eess.AS 82%

Audio-Visual Efficient Conformer for Robust Speech Recognition

Maxime Burchi, Radu Timofte

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.CL、eess.AS

2301.01134 2023-01-04 cs.MM cs.CL cs.CV 82%

Ring That Bell: A Corpus and Method for Multimodal Metaphor Detection in Videos

Khalid Alnajjar, Mika Hämäläinen, Shuo Zhang

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Figlang 2022

2212.14490 2023-01-02 cs.SD cs.CL cs.MM eess.AS 82%

Multi-modal deep learning system for depression and anxiety detection

Brian Diep, Marija Stanojevic, Jekaterina Novikova

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments accepted to the PAI4MH workshop at NeurIPS 2022

2212.04408 2022-12-09 cs.CV cs.AI cs.CL cs.LG 82%

OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Models

Jinze Bai, Rui Men, Hao Yang, Xuancheng Ren, Kai Dang, Yichang Zhang, Xiaohuan Zhou, Peng Wang, Sinan Tan, An Yang, Zeyu Cui, Yu Han, Shuai Bai, Wenbin Ge, Jianxin Ma, Junyang Lin, Jingren Zhou, Chang Zhou

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2212.01040 2022-12-05 cs.CV cs.MM cs.SD eess.AS 82%

Role of Audio in Audio-Visual Video Summarization

Ibrahim Shoer, Berkay Kopru, Engin Erzin

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

2212.00500 2022-12-02 cs.MM cs.CL cs.LG cs.SD eess.AS 82%

MMSpeech: Multi-modal Multi-task Encoder-Decoder Pre-training for Speech Recognition

Xiaohuan Zhou, Jiaming Wang, Zeyu Cui, Shiliang Zhang, Zhijie Yan, Jingren Zhou, Chang Zhou

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Submitted to ICASSP 2023

2211.04368 2022-11-09 cs.CL cs.CV cs.SD eess.AS 82%

A Multimodal Approach for Dementia Detection from Spontaneous Speech with Tensor Fusion Layer

Loukas Ilias, Dimitris Askounis, John Psarras

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI) - Oral Presentation

2211.02077 2022-11-07 cs.CV cs.AI cs.LG cs.MM 82%

Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization

Junru Wu, Yi Liang, Feng Han, Hassan Akbari, Zhangyang Wang, Cong Yu

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted at NeurIPS 2022

2209.04093 2022-10-27 cs.CV cs.MM cs.SD eess.AS 82%

Learning Audio-Visual embedding for Person Verification in the Wild

Peiwen Sun, Shanshan Zhang, Zishan Liu, Yougen Yuan, Taotao Zhang, Honggang Zhang, Pengfei Hu

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

2203.04114 2022-09-02 cs.MM cs.CV cs.SD eess.AS 82%

A study on joint modeling and data augmentation of multi-modalities for audio-visual scene classification

Qing Wang, Jun Du, Siyuan Zheng, Yunqing Li, Yajian Wang, Yuzhong Wu, Hu Hu, Chao-Han Huck Yang, Sabato Marco Siniscalchi, Yannan Wang, Chin-Hui Lee

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 5 pages, 1 figure

2204.03063 2022-08-29 cs.MM cs.CV cs.IR cs.SD eess.AS 82%

Late multimodal fusion for image and audio music transcription

María Alfaro-Contreras, Jose J. Valero-Mas, José M. Iñesta, Jorge Calvo-Zaragoza

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

2111.14448 2022-07-19 cs.CV cs.MM eess.AS 82%

AVA-AVD: Audio-Visual Speaker Diarization in the Wild

Eric Zhongcong Xu, Zeyang Song, Satoshi Tsutsui, Chao Feng, Mang Ye, Mike Zheng Shou

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments ACMMM 2022

2204.14057 2022-05-30 cs.SD cs.CV cs.MM eess.AS 82%

Unsupervised Voice-Face Representation Learning by Cross-Modal Prototype Contrast

Boqing Zhu, Kele Xu, Changjian Wang, Zheng Qin, Tao Sun, Huaimin Wang, Yuxing Peng

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 8 pages, 4 figures. Accepted by IJCAI-2022

2205.08383 2022-05-18 cs.LG cs.AI cs.CL cs.CV 82%

Bias and Fairness on Multimodal Emotion Detection Algorithms

Matheus Schmitz, Rehan Ahmed, Jimi Cao

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2205.05072 2022-05-11 cs.CV cs.MM cs.SD eess.AS 82%

Learning Visual Styles from Audio-Visual Associations

Tingle Li, Yichen Liu, Andrew Owens, Hang Zhao

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

2204.01977 2022-04-11 cs.SD cs.CV cs.MM eess.AS 82%

Audio-visual multi-channel speech separation, dereverberation and recognition

Guinan Li, Jianwei Yu, Jiajun Deng, Xunying Liu, Helen Meng

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by ICASSP 2022

2203.07990 2022-03-16 cs.MM cs.AI cs.CL 82%

UofA-Truth at Factify 2022 : Transformer And Transfer Learning Based Multi-Modal Fact-Checking

Abhishek Dhankar, Osmar R. Zaïane, Francois Bolduc

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

2203.02013 2022-03-07 cs.LG cs.AI cs.CL cs.CV 82%

DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations

Yiwei Lyu, Paul Pu Liang, Zihao Deng, Ruslan Salakhutdinov, Louis-Philippe Morency

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Code available at https://github.com/lvyiwei1/DIME

2108.05080 2022-03-02 cs.CV cs.MM cs.SD eess.AS 82%

FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset

Hasam Khalid, Shahroz Tariq, Minha Kim, Simon S. Woo

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Part of Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 2021)

2201.09165 2022-01-25 cs.MM cs.CV cs.SD eess.AS 82%

A Pre-trained Audio-Visual Transformer for Emotion Recognition

Minh Tran, Mohammad Soleymani

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by IEEE ICASSP 2022

2102.07594 2021-08-31 cs.CL cs.AI eess.AS 82%

Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT

Ye Bai, Jiangyan Yi, Jianhua Tao, Zhengkun Tian, Zhengqi Wen, Shuai Zhang

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 14 pages, 7 figures

2106.11059 2021-06-22 cs.LG 82%

Improving Multi-Modal Learning with Uni-Modal Teachers

Chenzhuang Du, Tingle Li, Yichen Liu, Zixin Wen, Tianyu Hua, Yue Wang, Hang Zhao

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)

2106.00451 2021-06-02 cs.CV cs.AI cs.CL 82%

Highlight Timestamp Detection Model for Comedy Videos via Multimodal Sentiment Analysis

Fan Huang

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2104.11116 2021-04-23 cs.CV cs.LG cs.MM cs.SD eess.AS eess.IV 82%

Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation

Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, Ziwei Liu

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. Code and models are available at https://github.com/Hangz-nju-cuhk/Talking-Face_PC-AVS

2005.04790 2021-04-09 cs.AI cs.CL cs.CV 82%

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes

Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, Davide Testuggine

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2020

1. 音频语音多模态 4578 篇

Contrastive Audio-Visual Masked Autoencoder

Vision Transformers are Parameter-Efficient Audio-Visual Learners

Contextually-rich human affect perception using multimodal scene information

Audio-Visual Segmentation

Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning

Audio-Visual Efficient Conformer for Robust Speech Recognition

Ring That Bell: A Corpus and Method for Multimodal Metaphor Detection in Videos

Multi-modal deep learning system for depression and anxiety detection

OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Models

Role of Audio in Audio-Visual Video Summarization

MMSpeech: Multi-modal Multi-task Encoder-Decoder Pre-training for Speech Recognition

A Multimodal Approach for Dementia Detection from Spontaneous Speech with Tensor Fusion Layer

Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization

Learning Audio-Visual embedding for Person Verification in the Wild

A study on joint modeling and data augmentation of multi-modalities for audio-visual scene classification

Late multimodal fusion for image and audio music transcription

AVA-AVD: Audio-Visual Speaker Diarization in the Wild

Unsupervised Voice-Face Representation Learning by Cross-Modal Prototype Contrast

Bias and Fairness on Multimodal Emotion Detection Algorithms

Learning Visual Styles from Audio-Visual Associations

Audio-visual multi-channel speech separation, dereverberation and recognition

UofA-Truth at Factify 2022 : Transformer And Transfer Learning Based Multi-Modal Fact-Checking

DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations

FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset

A Pre-trained Audio-Visual Transformer for Emotion Recognition

Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT

Improving Multi-Modal Learning with Uni-Modal Teachers

Highlight Timestamp Detection Model for Comedy Videos via Multimodal Sentiment Analysis

Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes