多模态大模型 - arXivDaily 专题

2205.00073 2022-05-03 cs.CV 79%

On Negative Sampling for Audio-Visual Contrastive Learning from Movies

Mahdi M. Kalayeh, Shervin Ardeshir, Lingyi Liu, Nagendra Kamath, Ashok Chandrashekar

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV

Comments arXiv admin note: substantial text overlap with arXiv:2106.08513

2204.12366 2022-04-27 cs.MM 79%

Robust Audio-Visual Instance Discrimination via Active Contrastive Set Mining

Hanyu Xuan, Yihong Xu, Shuo Chen, Zhiliang Wu, Jian Yang, Yan Yan, Xavier Alameda-Pineda

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.MM

Comments 7 pages, 4 figures, accepted at IJCAI 2022

2204.08686 2022-04-21 cs.SD eess.AS 79%

Audio-Visual Wake Word Spotting System For MISP Challenge 2021

Yanguang Xu, Jianwei Sun, Yang Han, Shuaijiang Zhao, Chaoyang Mei, Tingwei Guo, Shuran Zhou, Chuandong Xie, Wei Zou, Xiangang Li, Shuran Zhou, Chuandong Xie, Wei Zou, Xiangang Li

专题命中音频语音多模态：audio-visual(title)；multimodal(abstract)；分类 eess.AS

Comments Accepted to ICASSP 2022

2110.10101 2022-04-21 cs.CV 79%

Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognition

Mirco Planamente, Chiara Plizzari, Emanuele Alberti, Barbara Caputo

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV

Comments Accepted at WACV 2022. arXiv admin note: substantial text overlap with arXiv:2106.01689

2202.04303 2022-04-20 cs.LG cs.SD eess.AS 79%

TinyM$^2$Net: A Flexible System Algorithm Co-designed Multimodal Learning Framework for Tiny Devices

Hasib-Al Rashid, Pretom Roy Ovi, Carl Busart, Aryya Gangopadhyay, Tinoosh Mohsenin

专题命中音频语音多模态：multimodal(title,abstract)；分类 eess.AS

Comments tinyML Research Symposium 2022

2204.02905 2022-04-07 cs.CL cs.HC 79%

EMMT: A simultaneous eye-tracking, 4-electrode EEG and audio corpus for multi-modal reading and translation scenarios

Sunit Bhattacharya, Věra Kloudová, Vilém Zouhar, Ondřej Bojar

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CL

Comments Submitted to Nature Scientific Data

2203.13032 2022-04-01 cs.CV eess.IV 79%

Multi-modal Emotion Estimation for in-the-wild Videos

Liyu Meng, Yuchen Liu, Xiaolong Liu, Zhaopei Huang, Yuan Cheng, Meng Wang, Chuanhe Liu, Qin Jin

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CV

2203.00674 2022-03-02 cs.CL cs.CY 79%

Advancing an Interdisciplinary Science of Conversation: Insights from a Large Multimodal Corpus of Human Speech

Andrew Reece, Gus Cooney, Peter Bull, Christine Chung, Bryn Dawson, Casey Fitzpatrick, Tamara Glazer, Dean Knox, Alex Liebscher, Sebastian Marin

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL

Comments 116 pages; Cross-posted to PsyArxiv; To request data access, go to https://betterup-data-requests.herokuapp.com/

2202.10976 2022-02-23 cs.SD eess.AS 79%

DRVC: A Framework of Any-to-Any Voice Conversion with Self-Supervised Learning

Qiqi Wang, Xulong Zhang, Jianzong Wang, Ning Cheng, Jing Xiao

专题命中音频语音多模态：any-to-any(title,abstract)；分类 eess.AS

Comments Published at ICASSP 2022

2112.15110 2022-02-23 cs.SD cs.LG eess.AS 79%

Audio-to-symbolic Arrangement via Cross-modal Music Representation Learning

Ziyu Wang, Dejing Xu, Gus Xia, Ying Shan

专题命中音频语音多模态：cross-modal(title,abstract)；分类 eess.AS

2202.09195 2022-02-21 cs.LG cs.MM 79%

A Review on Methods and Applications in Multimodal Deep Learning

Jabeen Summaira, Xi Li, Amin Muhammad Shoib, Jabbar Abdul

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.MM

Comments 29 pages. arXiv admin note: substantial text overlap with arXiv:2105.11087

Journal ref ACM Transactions on Multimedia Computing, Communications, and Applications 2022

2202.08974 2022-02-21 cs.SD cs.HC cs.LG cs.RO eess.AS 79%

Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models

Sarala Padi, Seyed Omid Sadjadi, Dinesh Manocha, Ram D. Sriram

专题命中音频语音多模态：multimodal(title,abstract)；分类 eess.AS

Comments arXiv admin note: substantial text overlap with arXiv:2108.02510

2202.07360 2022-02-16 cs.HC cs.CV 79%

Multimodal Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the Vehicle

Abdul Rafey Aftab, Michael von der Beeck

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV

Journal ref 27th International Conference on Intelligent User Interfaces (IUI '22), March 22--25, 2022, Helsinki, Finland

2202.06507 2022-02-15 eess.AS cs.LG cs.SD q-bio.QM 79%

EMGSE: Acoustic/EMG Fusion for Multimodal Speech Enhancement

Kuan-Chen Wang, Kai-Chun Liu, Hsin-Min Wang, Yu Tsao

专题命中音频语音多模态：multimodal(title,abstract)；分类 eess.AS

Comments 5 pages, 4 figures, and 3 tables

2202.06218 2022-02-15 cs.LG cs.CL 79%

Emotion Based Hate Speech Detection using Multimodal Learning

Aneri Rana, Sonali Jha

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL

2202.05961 2022-02-15 cs.CV eess.IV 79%

Audio-Visual Fusion Layers for Event Type Aware Video Recognition

Arda Senocak, Junsik Kim, Tae-Hyun Oh, Hyeonggon Ryu, Dingzeyu Li, In So Kweon

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV

2112.00158 2022-01-28 eess.AS 79%

Representation learning through cross-modal conditional teacher-student training for speech emotion recognition

Sundararajan Srinivasan, Zhaocheng Huang, Katrin Kirchhoff

专题命中音频语音多模态：cross-modal(title)；multimodal(abstract)；分类 eess.AS

Comments Accepted for publication at IEEE ICASSP 2022

2112.14088 2021-12-30 cs.CV 79%

Synchronized Audio-Visual Frames with Fractional Positional Encoding for Transformers in Video-to-Text Translation

Philipp Harzig, Moritz Einfalt, Rainer Lienhart

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV

2007.04954 2021-12-30 cs.CV cs.GR cs.LG cs.RO 79%

ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation

Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Michael Lingelbach, Aidan Curtis, Kevin Feigelis, Daniel M. Bear, Dan Gutfreund, David Cox, Antonio Torralba, James J. DiCarlo, Joshua B. Tenenbaum, Josh H. McDermott, Daniel L. K. Yamins

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CV

Comments Oral Presentation at NeurIPS 21 Datasets and Benchmarks Track. Project page: http://www.threedworld.org

2112.12409 2021-12-24 cs.CV 79%

InstaIndoor and Multi-modal Deep Learning for Indoor Scene Recognition

Andreea Glavan, Estefania Talavera

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CV

2112.09172 2021-12-20 cs.CV cs.LG eess.IV 79%

An Audio-Visual Dataset and Deep Learning Frameworks for Crowded Scene Classification

Lam Pham, Dat Ngo, Phu X. Nguyen, Truong Hoang, Alexander Schindler

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV

2112.02749 2021-12-07 cs.CV 79%

One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning

Suzhen Wang, Lincheng Li, Yu Ding, Xin Yu

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV

Comments Accepted by AAAI 2022

Journal ref AAAI 2022

2106.15909 2021-11-30 eess.AS cs.SD 79%

Effect of acoustic scene complexity and visual scene representation on auditory perception in virtual audio-visual environments

Stefan Fichna, Thomas Biberger, Bernhard U. Seeber, Stephan D. Ewert

专题命中音频语音多模态：audio-visual(title,abstract)；分类 eess.AS

Comments Accepted publication in Proceedings of 3DA 2021 International Conference on Immersive and 3D Audio

2111.08702 2021-11-18 cs.CV cs.CR 79%

The Multiscenario Multienvironment BioSecure Multimodal Database (BMDB)

Javier Ortega-Garcia, Julian Fierrez, Fernando Alonso-Fernandez, Javier Galbally, Manuel R Freire, Joaquin Gonzalez-Rodriguez, Carmen Garcia-Mateo, Jose-Luis Alba-Castro, Elisardo Gonzalez-Agulla, Enrique Otero-Muras, Sonia Garcia-Salicetti, Lorene Allano, Bao Ly-Van, Bernadette Dorizzi, Josef Kittler, Thirimachos Bourlai, Norman Poh, Farzin Deravi, Ming NR Ng, Michael Fairhurst, Jean Hennebert, Andreas Humm, Massimo Tistarelli, Linda Brodo, Jonas Richiardi, Andrezj Drygajlo, Harald Ganster, Federico M Sukno, Sri-Kaushik Pavani, Alejandro Frangi, Lale Akarun, Arman Savran

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV

Comments Published at IEEE Transactions on Pattern Analysis and Machine Intelligence journal

2111.03472 2021-11-08 cs.CR cs.CV eess.IV 79%

BiosecurID: a multimodal biometric database

Julian Fierrez, Javier Galbally, Javier Ortega-Garcia, Manuel R Freire, Fernando Alonso-Fernandez, Daniel Ramos, Doroteo Torre Toledano, Joaquin Gonzalez-Rodriguez, Juan A Siguenza, Javier Garrido-Salas, E Anguiano, Guillermo Gonzalez-de-Rivera, Ricardo Ribalda, Marcos Faundez-Zanuy, JA Ortega, Valentín Cardeñoso-Payo, A Viloria, Carlos E Vivaracho, Q Isaac Moro, Juan J Igarza, J Sanchez, Inmaculada Hernaez, Carlos Orrite-Urunuela, Francisco Martinez-Contreras, Juan José Gracia-Roche

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV

Comments Published at Pattern Analysis and Applications journal

2104.13276 2021-11-01 cs.SD cs.DB cs.IR cs.LG eess.AS 79%

MULTIMODAL ANALYSIS: Informed content estimation and audio source separation

Gabriel Meseguer-Brocal

专题命中音频语音多模态：multimodal(title,abstract)；分类 eess.AS

Comments Ph.D. dissertation. Thesis supervisor: Geoffroy Peeters. Jury:Laurent Girin, Gaël Richard, Rachel Bittner, Elena Cabrio, Bruno Gas, Perfecto Herrera Boyer, Antoine Liutkus

2110.13570 2021-10-28 cs.CV 79%

Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition

Siyang Song, Zilong Shao, Shashank Jaiswal, Linlin Shen, Michel Valstar, Hatice Gunes

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV

Comments Submitted to IJCV

2110.02500 2021-10-07 eess.AS 79%

MediumVC: Any-to-any voice conversion using synthetic specific-speaker speeches as intermedium features

Yewei Gu, Zhenyu Zhang, Xiaowei Yi, Xianfeng Zhao

专题命中音频语音多模态：any-to-any(title,abstract)；分类 eess.AS

Comments 5 pages, 2 figures

2109.09536 2021-09-21 cs.CV cs.LG 79%

Audio-Visual Speech Recognition is Worth 32$\times$32$\times$8 Voxels

Dmitriy Serdyuk, Otavio Braga, Olivier Siohan

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV

Comments 7 pages, 2 figures, 4 tables. A draft for a paper accepted to ASRU workshop

2109.00181 2021-09-02 cs.SD cs.AI 79%

CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations

Hang Li, Yu Kang, Tianqiao Liu, Wenbiao Ding, Zitao Liu

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.AI

Comments The 2021 Conference on Empirical Methods in Natural Language Processing

1. 音频语音多模态 4587 篇

On Negative Sampling for Audio-Visual Contrastive Learning from Movies

Robust Audio-Visual Instance Discrimination via Active Contrastive Set Mining

Audio-Visual Wake Word Spotting System For MISP Challenge 2021

Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognition

TinyM$^2$Net: A Flexible System Algorithm Co-designed Multimodal Learning Framework for Tiny Devices

EMMT: A simultaneous eye-tracking, 4-electrode EEG and audio corpus for multi-modal reading and translation scenarios

Multi-modal Emotion Estimation for in-the-wild Videos

Advancing an Interdisciplinary Science of Conversation: Insights from a Large Multimodal Corpus of Human Speech

DRVC: A Framework of Any-to-Any Voice Conversion with Self-Supervised Learning

Audio-to-symbolic Arrangement via Cross-modal Music Representation Learning

A Review on Methods and Applications in Multimodal Deep Learning

Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models

Multimodal Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the Vehicle

EMGSE: Acoustic/EMG Fusion for Multimodal Speech Enhancement

Emotion Based Hate Speech Detection using Multimodal Learning

Audio-Visual Fusion Layers for Event Type Aware Video Recognition

Representation learning through cross-modal conditional teacher-student training for speech emotion recognition

Synchronized Audio-Visual Frames with Fractional Positional Encoding for Transformers in Video-to-Text Translation

ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation

InstaIndoor and Multi-modal Deep Learning for Indoor Scene Recognition

An Audio-Visual Dataset and Deep Learning Frameworks for Crowded Scene Classification

One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning

Effect of acoustic scene complexity and visual scene representation on auditory perception in virtual audio-visual environments

The Multiscenario Multienvironment BioSecure Multimodal Database (BMDB)

BiosecurID: a multimodal biometric database

MULTIMODAL ANALYSIS: Informed content estimation and audio source separation

Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition

MediumVC: Any-to-any voice conversion using synthetic specific-speaker speeches as intermedium features

Audio-Visual Speech Recognition is Worth 32$\times$32$\times$8 Voxels

CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations