多模态大模型 - arXivDaily 专题

2401.05698 2024-04-02 cs.CV cs.HC cs.MM cs.SD eess.AS 82%

HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition

Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by Information Fusion. The code is available at https://github.com/sunlicai/HiCMAE

Journal ref Information Fusion, 2024

2403.18821 2024-03-28 cs.SD cs.CV cs.MM eess.AS 82%

Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark

Ziyang Chen, Israel D. Gebru, Christian Richardt, Anurag Kumar, William Laney, Andrew Owens, Alexander Richard

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to CVPR 2024. Project site: https://facebookresearch.github.io/real-acoustic-fields/

2309.10787 2024-03-20 eess.AS cs.CV cs.MM cs.SD 82%

AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models

Yuan Tseng, Layne Berry, Yi-Ting Chen, I-Hsiang Chiu, Hsuan-Hao Lin, Max Liu, Puyuan Peng, Yi-Jen Shih, Hung-Yu Wang, Haibin Wu, Po-Yao Huang, Chun-Mao Lai, Shang-Wen Li, David Harwath, Yu Tsao, Shinji Watanabe, Abdelrahman Mohamed, Chi-Luen Feng, Hung-yi Lee

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to ICASSP 2024; Evaluation Code: https://github.com/roger-tseng/av-superb Submission Platform: https://av.superbbenchmark.org

2403.03145 2024-03-06 cs.CV cs.LG cs.MM cs.SD eess.AS 82%

Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for Audio-Visual Source Localization

Yuxin Guo, Shijie Ma, Hu Su, Zhiqing Wang, Yuhao Zhao, Wei Zou, Siyang Sun, Yun Zheng

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to NeurIPS2023

2403.03095 2024-03-06 cs.CV cs.MM cs.SD eess.AS 82%

Cross Pseudo-Labeling for Semi-Supervised Audio-Visual Source Localization

Yuxin Guo, Shijie Ma, Yuhao Zhao, Hu Su, Wei Zou

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted To ICASSP2024

2402.06984 2024-02-13 cs.SD cs.CV cs.MM eess.AS eess.IV 82%

Speech motion anomaly detection via cross-modal translation of 4D motion fields from tagged MRI

Xiaofeng Liu, Fangxu Xing, Jiachen Zhuo, Maureen Stone, Jerry L. Prince, Georges El Fakhri, Jonghye Woo

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments SPIE Medical Imaging 2024: Image Processing

2305.12793 2024-02-06 eess.AS cs.CL cs.MM cs.SD 82%

Zero-Shot End-to-End Spoken Language Understanding via Cross-Modal Selective Self-Training

Jianfeng He, Julian Salazar, Kaisheng Yao, Haoqi Li, Jinglun Cai

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments 18 pages, 7 figures

2309.07929 2024-02-05 cs.CV cs.LG cs.MM cs.SD eess.AS 82%

Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer

Yaoting Wang, Weisong Liu, Guangyao Li, Jian Ding, Di Hu, Xi Li

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by AAAI 2024

2302.06419 2024-01-23 eess.AS cs.AI cs.CL 82%

AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations

Jiachen Lian, Alexei Baevski, Wei-Ning Hsu, Michael Auli

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 2023 ASRU

2306.04047 2023-12-29 cs.CV cs.AI cs.CL cs.LG 82%

CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments

Xiulong Liu, Sudipta Paul, Moitreya Chatterjee, Anoop Cherian

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at AAAI 2024

2312.15197 2023-12-27 cs.SD cs.CL cs.CV eess.AS 82%

TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation

Xize Cheng, Rongjie Huang, Linjun Li, Tao Jin, Zehan Wang, Aoxiong Yin, Minglei Li, Xinyu Duan, changpeng yang, Zhou Zhao

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.CL、eess.AS

2305.07223 2023-12-27 cs.SD cs.CV cs.MM eess.AS 82%

Transavs: End-To-End Audio-Visual Segmentation With Transformer

Yuhang Ling, Yuxi Li, Zhenye Gan, Jiangning Zhang, Mingmin Chi, Yabiao Wang

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 4 pages, 3 figures

2306.09126 2023-11-15 cs.SD cs.CV cs.MM eess.AS eess.IV 82%

STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Events

Kazuki Shimada, Archontis Politis, Parthasaarathy Sudarsanam, Daniel Krause, Kengo Uchida, Sharath Adavanne, Aapo Hakala, Yuichiro Koyama, Naoya Takahashi, Shusuke Takahashi, Tuomas Virtanen, Yuki Mitsufuji

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 27 pages, 9 figures, accepted for publication in NeurIPS 2023 Track on Datasets and Benchmarks

2305.13507 2023-10-27 cs.CL cs.AI cs.CV 82%

Multimodal Automated Fact-Checking: A Survey

Mubashara Akhtar, Michael Schlichtkrull, Zhijiang Guo, Oana Cocarascu, Elena Simperl, Andreas Vlachos

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP): Findings

2305.11019 2023-10-10 cs.CV cs.AI cs.MM 82%

Annotation-free Audio-Visual Segmentation

Jinxiang Liu, Yu Wang, Chen Ju, Chaofan Ma, Ya Zhang, Weidi Xie

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Camera-ready version for WACV 2024; project page is https://jinxiang-liu.github.io/anno-free-AVS/

2309.14320 2023-09-26 cs.RO 82%

MUTEX: Learning Unified Policies from Multimodal Task Specifications

Rutav Shah, Roberto Martín-Martín, Yuke Zhu

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)

Comments Accepted at 7th Conference on Robot Learning (CoRL 2023), Atlanta, USA

2309.13942 2023-09-26 cs.CV cs.MM cs.SD eess.AS 82%

Speed Co-Augmentation for Unsupervised Audio-Visual Pre-training

Jiangliu Wang, Jianbo Jiao, Yibing Song, Stephen James, Zhan Tong, Chongjian Ge, Pieter Abbeel, Yun-hui Liu

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Published at the CVPR 2023 Sight and Sound workshop

2303.16342 2023-09-26 cs.CV cs.AI cs.CL 82%

Language-Guided Audio-Visual Source Separation via Trimodal Consistency

Reuben Tan, Arijit Ray, Andrea Burns, Bryan A. Plummer, Justin Salamon, Oriol Nieto, Bryan Russell, Kate Saenko

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at CVPR 2023

2309.07925 2023-09-18 eess.AS cs.AI cs.MM cs.SD 82%

Hierarchical Audio-Visual Information Fusion with Multi-label Joint Decoding for MER 2023

Haotian Wang, Yuxuan Xi, Hang Chen, Jun Du, Yan Song, Qing Wang, Hengshun Zhou, Chenxi Wang, Jiefeng Ma, Pengfei Hu, Ya Jiang, Shi Cheng, Jie Zhang, Yuzhe Weng

专题命中音频语音多模态：audio-visual(title)；multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 5 pages, 4 figures

Journal ref The 31st ACM International Conference on Multimedia (MM'23), 2023

2308.06087 2023-08-21 cs.MM cs.AI cs.CV 82%

Audio-Visual Spatial Integration and Recursive Attention for Robust Sound Source Localization

Sung Jin Um, Dongjin Kim, Jung Uk Kim

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Camera-Ready, ACM MM 2023

2307.15167 2023-07-31 cs.HC 82%

PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data

Zheng Zhang, Zheng Ning, Chenliang Xu, Yapeng Tian, Toby Jia-Jun Li

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)

Comments 18 pages, published in UIST'23

2305.16049 2023-07-31 cs.CV cs.MM cs.SD eess.AS 82%

CN-Celeb-AV: A Multi-Genre Audio-Visual Dataset for Person Recognition

Lantian Li, Xiaolou Li, Haoyu Jiang, Chen Chen, Ruihai Hou, Dong Wang

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments INTERSPEECH 2023

2306.10563 2023-06-21 eess.AS cs.CV cs.MM cs.SD 82%

Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition

Yuchen Hu, Ruizhe Li, Chen Chen, Chengwei Qin, Qiushi Zhu, Eng Siong Chng

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 19 pages, 9 figures, Accepted by ACL 2023

2306.09093 2023-06-16 cs.CL cs.AI cs.CV 82%

Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration

Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, Zhaopeng Tu

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Longyue Wang is the corresponding author. Our project page is at https://github.com/lyuchenyang/Macaw-LLM

2303.07667 2023-06-13 cs.SD 82%

Improving Music Genre Classification from Multi-Modal Properties of Music and Genre Correlations Perspective

Ganghui Ru, Xulong Zhang, Jianzong Wang, Ning Cheng, Jing Xiao

专题命中音频语音多模态：multi-modal(title,abstract)；cross-modal(abstract)

Comments Accepted by ICASSP 2023

2305.19458 2023-06-01 cs.SD cs.CV cs.LG cs.MM eess.AS 82%

A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition

Shentong Mo, Pedro Morgado

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

2305.14635 2023-05-26 cs.CL cs.AI cs.SD eess.AS 82%

CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation

Yan Zhou, Qingkai Fang, Yang Feng

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments ACL 2023 main conference

2305.04476 2023-05-25 eess.AS cs.CL cs.MM cs.SD 82%

AlignSTS: Speech-to-Singing Conversion via Cross-Modal Alignment

Ruiqi Li, Rongjie Huang, Lichao Zhang, Jinglin Liu, Zhou Zhao

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Findings of ACL 2023

2305.02814 2023-05-05 cs.MM cs.AI cs.CV cs.HC 82%

Noise-Resistant Multimodal Transformer for Emotion Recognition

Yuanyuan Liu, Haoyu Zhang, Yibing Zhan, Zijing Chen, Guanghao Yin, Lin Wei, Zhe Chen

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

2211.03643 2023-05-04 cs.SD cs.CL cs.MM eess.AS 82%

Egocentric Audio-Visual Noise Suppression

Roshan Sharma, Weipeng He, Ju Lin, Egor Lakomkin, Yang Liu, Kaustubh Kalgaonkar

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted to ICASSP 2023

1. 音频语音多模态 4578 篇

HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition

Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark

AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models

Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for Audio-Visual Source Localization

Cross Pseudo-Labeling for Semi-Supervised Audio-Visual Source Localization

Speech motion anomaly detection via cross-modal translation of 4D motion fields from tagged MRI

Zero-Shot End-to-End Spoken Language Understanding via Cross-Modal Selective Self-Training

Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer

AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations

CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments

TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation

Transavs: End-To-End Audio-Visual Segmentation With Transformer

STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Events

Multimodal Automated Fact-Checking: A Survey

Annotation-free Audio-Visual Segmentation

MUTEX: Learning Unified Policies from Multimodal Task Specifications

Speed Co-Augmentation for Unsupervised Audio-Visual Pre-training

Language-Guided Audio-Visual Source Separation via Trimodal Consistency

Hierarchical Audio-Visual Information Fusion with Multi-label Joint Decoding for MER 2023

Audio-Visual Spatial Integration and Recursive Attention for Robust Sound Source Localization

PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data

CN-Celeb-AV: A Multi-Genre Audio-Visual Dataset for Person Recognition

Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition

Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration

Improving Music Genre Classification from Multi-Modal Properties of Music and Genre Correlations Perspective

A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition

CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation

AlignSTS: Speech-to-Singing Conversion via Cross-Modal Alignment

Noise-Resistant Multimodal Transformer for Emotion Recognition

Egocentric Audio-Visual Noise Suppression