多模态大模型 - arXivDaily 专题

2212.04679 2023-09-21 cs.CV 83%

Motion and Context-Aware Audio-Visual Conditioned Video Prediction

Yating Xu, Conghui Hu, Gim Hee Lee

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments BMVC 2023

2309.10077 2023-09-20 cs.LG cs.AI 83%

GAME: Generalized deep learning model towards multimodal data integration for early screening of adolescent mental disorders

Zhicheng Du, Chenyao Jiang, Xi Yuan, Shiyao Zhai, Zhengyang Lei, Shuyue Ma, Yang Liu, Qihui Ye, Chufan Xiao, Qiming Huang, Ming Xu, Dongmei Yu, Peiwu Qin

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI

2305.16406 2023-08-24 cs.CL 83%

Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech

Loukas Ilias, Dimitris Askounis

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL

Comments Knowledge-Based Systems

Journal ref Knowledge-Based Systems (Volume: 277, October 2023)

2308.10175 2023-08-22 cs.CV 83%

BAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledge

Chen Liu, Peike Li, Hu Zhang, Lincheng Li, Zi Huang, Dadong Wang, Xin Yu

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

2307.14682 2023-07-28 cs.CV 83%

Unified Adversarial Patch for Visible-Infrared Cross-modal Attacks in the Physical World

Xingxing Wei, Yao Huang, Yitong Sun, Jie Yu

专题命中音频语音多模态：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments 13 pages, 16 figures. arXiv admin note: substantial text overlap with arXiv:2307.07859

2307.11519 2023-07-24 cs.AI cs.CL cs.CV cs.LG cs.SD eess.AS 83%

Multi-modal Hate Speech Detection using Machine Learning

Fariha Tahosin Boishakhi, Ponkoj Chandra Shill, Md. Golam Rabiul Alam

专题命中音频语音多模态：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 5 pages, 2 figures, conference

2307.10802 2023-07-21 cs.CV cs.AI cs.CL cs.LG cs.MM 83%

Meta-Transformer: A Unified Framework for Multimodal Learning

Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, Xiangyu Yue

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project website: https://kxgong.github.io/meta_transformer/

2307.07859 2023-07-20 cs.CV 83%

Unified Adversarial Patch for Cross-modal Attacks in the Physical World

Xingxing Wei, Yao Huang, Yitong Sun, Jie Yu

专题命中音频语音多模态：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments 10 pages, 8 figures, accepted by ICCV2023

2305.01979 2023-07-18 cs.CV 83%

Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localization

Zhixi Cai, Shreya Ghosh, Abhinav Dhall, Tom Gedeon, Kalin Stefanov, Munawar Hayat

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments The paper is under consideration/review at Computer Vision and Image Understanding Journal

2307.07277 2023-07-17 cs.CL 83%

Are words equally surprising in audio and audio-visual comprehension?

Pranava Madhyastha, Ye Zhang, Gabriella Vigliocco

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CL

Comments In CogSci 2023

2306.16034 2023-06-29 cs.AI cs.NI 83%

Stone Needle: A General Multimodal Large-scale Model Framework towards Healthcare

Weihua Liu, Yong Zuo

专题命中音频语音多模态：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.AI

2306.15644 2023-06-28 cs.CL 83%

Style-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos

Chiori Hori, Puyuan Peng, David Harwath, Xinyu Liu, Kei Ota, Siddarth Jain, Radu Corcodel, Devesh Jha, Diego Romeres, Jonathan Le Roux

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CL

Comments Accepted to Interspeech2023

2306.09431 2023-06-19 cs.MM 83%

Towards Long Form Audio-visual Video Understanding

Wenxuan Hou, Guangyao Li, Yapeng Tian, Di Hu

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.MM

2306.02263 2023-06-06 cs.SD cs.CV 83%

MAVD: The First Open Large-Scale Mandarin Audio-Visual Dataset with Depth Information

Jianrong Wang, Yuchen Huo, Li Liu, Tianyi Xu, Qi Li, Sen Li

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

2305.11000 2023-05-22 cs.CL 83%

SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities

Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, Xipeng Qiu

专题命中音频语音多模态：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CL

Comments work in progress

2304.12082 2023-04-25 cs.SD eess.AS 83%

Deep Audio-Visual Singing Voice Transcription based on Self-Supervised Learning Models

Xiangming Gu, Wei Zeng, Jianan Zhang, Longshen Ou, Ye Wang

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 eess.AS

2303.14768 2023-03-28 cs.CV 83%

Collaborative Noisy Label Cleaner: Learning Scene-aware Trailers for Multi-modal Highlight Detection in Movies

Bei Gan, Xiujun Shu, Ruizhi Qiao, Haoqian Wu, Keyu Chen, Hanjun Li, Bo Ren

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV

Comments Accepted to CVPR2023

2204.13386 2023-03-21 cs.CV 83%

Self-supervised Contrastive Learning for Audio-Visual Action Recognition

Yang Liu, Ying Tan, Haoyuan Lan

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments 6 pages, 2 figures

2303.06341 2023-03-14 eess.AS 83%

The NPU-ASLP System for Audio-Visual Speech Recognition in MISP 2022 Challenge

Pengcheng Guo, He Wang, Bingshen Mu, Ao Zhang, Peikun Chen

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 eess.AS

Comments 2 pages, accepted by ICASSP 2023

2303.06129 2023-03-13 cs.CV 83%

Single-branch Network for Multimodal Training

Muhammad Saad Saeed, Shah Nawaz, Muhammad Haris Khan, Muhammad Zaigham Zaheer, Karthik Nandakumar, Muhammad Haroon Yousaf, Arif Mahmood

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted at ICASSP 2023

2301.09595 2023-02-23 cs.CV 83%

Zorro: the masked multimodal transformer

Adrià Recasens, Jason Lin, Joāo Carreira, Drew Jaegle, Luyu Wang, Jean-baptiste Alayrac, Pauline Luc, Antoine Miech, Lucas Smaira, Ross Hemsley, Andrew Zisserman

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV

2302.07702 2023-02-16 cs.CV 83%

Audio-Visual Contrastive Learning with Temporal Self-Supervision

Simon Jenni, Alexander Black, John Collomosse

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments AAAI-23

2302.05727 2023-02-14 cs.CV 83%

Flexible-modal Deception Detection with Audio-Visual Adapter

Zhaoxu Li, Zitong Yu, Nithish Muthuchamy Selvaraj, Xiaobao Guo, Bingquan Shen, Adams Wai-Kin Kong, Alex Kot

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

2302.02447 2023-02-07 eess.AS cs.SD 83%

cross-modal fusion techniques for utterance-level emotion recognition from text and speech

Jiachen Luo, Huy Phan, Joshua Reiss

专题命中音频语音多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 eess.AS

Comments 6 pages, 2 figures

2107.07268 2022-12-13 cs.MM cs.IR 83%

Cross-modal Variational Auto-encoder for Content-based Micro-video Background Music Recommendation

Jing Yi, Yaochen Zhu, Jiayi Xie, Zhenzhong Chen

专题命中音频语音多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.MM

2211.10883 2022-11-22 cs.CV 83%

Audio-visual video face hallucination with frequency supervision and cross modality support by speech based lip reading loss

Shailza Sharma, Abhinav Dhall, Vinay Kumar, Vivek Singh Bawa

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV

2205.12216 2022-11-11 cs.CL 83%

T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation

Paul-Ambroise Duquenne, Hongyu Gong, Benoît Sagot, Holger Schwenk

专题命中音频语音多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CL

2210.15903 2022-10-31 eess.AS cs.SD eess.SP 83%

Speaker recognition with two-step multi-modal deep cleansing

Ruijie Tao, Kong Aik Lee, Zhan Shi, Haizhou Li

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 eess.AS

Comments 5 pages, 3 figures

2210.14581 2022-10-27 eess.AS cs.SD eess.IV 83%

Deep Learning Based Audio-Visual Multi-Speaker DOA Estimation Using Permutation-Free Loss Function

Qing Wang, Hang Chen, Ya Jiang, Zhe Wang, Yuyang Wang, Jun Du, Chin-Hui Lee

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 eess.AS

Comments 5 pages, 3 figures, accepted by ISCSLP 2022

2209.09068 2022-09-20 cs.CV 83%

Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention

R Gnana Praveen, Eric Granger, Patrick Cardinal

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments arXiv admin note: substantial text overlap with arXiv:2203.14779, arXiv:2111.05222

1. 音频语音多模态 4567 篇

Motion and Context-Aware Audio-Visual Conditioned Video Prediction

GAME: Generalized deep learning model towards multimodal data integration for early screening of adolescent mental disorders

Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech

BAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledge

Unified Adversarial Patch for Visible-Infrared Cross-modal Attacks in the Physical World

Multi-modal Hate Speech Detection using Machine Learning

Meta-Transformer: A Unified Framework for Multimodal Learning

Unified Adversarial Patch for Cross-modal Attacks in the Physical World

Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localization

Are words equally surprising in audio and audio-visual comprehension?

Stone Needle: A General Multimodal Large-scale Model Framework towards Healthcare

Style-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos

Towards Long Form Audio-visual Video Understanding

MAVD: The First Open Large-Scale Mandarin Audio-Visual Dataset with Depth Information

SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities

Deep Audio-Visual Singing Voice Transcription based on Self-Supervised Learning Models

Collaborative Noisy Label Cleaner: Learning Scene-aware Trailers for Multi-modal Highlight Detection in Movies

Self-supervised Contrastive Learning for Audio-Visual Action Recognition

The NPU-ASLP System for Audio-Visual Speech Recognition in MISP 2022 Challenge

Single-branch Network for Multimodal Training

Zorro: the masked multimodal transformer

Audio-Visual Contrastive Learning with Temporal Self-Supervision

Flexible-modal Deception Detection with Audio-Visual Adapter

cross-modal fusion techniques for utterance-level emotion recognition from text and speech

Cross-modal Variational Auto-encoder for Content-based Micro-video Background Music Recommendation

Audio-visual video face hallucination with frequency supervision and cross modality support by speech based lip reading loss

T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation

Speaker recognition with two-step multi-modal deep cleansing

Deep Learning Based Audio-Visual Multi-Speaker DOA Estimation Using Permutation-Free Loss Function

Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention