多模态大模型 - arXivDaily 专题

2102.06269 2021-02-15 eess.IV cs.SD eess.AS 83%

Disentanglement for audio-visual emotion recognition using multitask setup

Raghuveer Peri, Srinivas Parthasarathy, Charles Bradshaw, Shiva Sundaram

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 eess.AS

Comments Accepted for ICASSP 2021, 5 pages

2102.01326 2021-02-03 eess.AS cs.LG cs.SD 83%

Multimodal Attention Fusion for Target Speaker Extraction

Hiroshi Sato, Tsubasa Ochiai, Keisuke Kinoshita, Marc Delcroix, Tomohiro Nakatani, Shoko Araki

专题命中音频语音多模态：multimodal(title)；multi-modal(abstract)；audio-visual(abstract)；分类 eess.AS

Comments 7 pages, 5 figures

Journal ref in IEEE Spoken Language Technology Workshop (SLT), 2021, pp. 778-784

2012.00063 2020-12-02 eess.AS cs.SD eess.IV 83%

Detecting expressions with multimodal transformers

Srinivas Parthasarathy, Shiva Sundaram

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 eess.AS

Comments IEEE Spoken Language Technology Workshop 2021

2006.16228 2020-11-02 cs.CV 83%

Self-Supervised MultiModal Versatile Networks

Jean-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, Andrew Zisserman

专题命中音频语音多模态：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments To appear in the Thirty-Fourth Annual Conference on Neural Information Processing Systems (NeurIPS 2020)

2003.07032 2020-10-26 eess.AS cs.SD eess.IV 83%

Multi-modal Multi-channel Target Speech Separation

Rongzhi Gu, Shi-Xiong Zhang, Yong Xu, Lianwu Chen, Yuexian Zou, Dong Yu

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 eess.AS

Comments accepted in IEEE Journal of Selcted Topics in Signal Processing

2010.10839 2020-10-22 cs.CL 83%

TMT: A Transformer-based Modal Translator for Improving Multimodal Sequence Representations in Audio Visual Scene-aware Dialog

Wubo Li, Dongwei Jiang, Wei Zou, Xiangang Li

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CL

2009.02095 2020-10-02 eess.AS cs.LG cs.SD 83%

SEANet: A Multi-modal Speech Enhancement Network

Marco Tagliasacchi, Yunpeng Li, Karolis Misiunas, Dominik Roblek

专题命中音频语音多模态：multi-modal(title,abstract)；multimodal(abstract)；分类 eess.AS

Comments Accepted to INTERSPEECH 2020

2009.09561 2020-09-22 cs.SD cs.LG eess.AS 83%

Correlating Subword Articulation with Lip Shapes for Embedding Aware Audio-Visual Speech Enhancement

Hang Chen, Jun Du, Yu Hu, Li-Rong Dai, Bao-Cai Yin, Chin-Hui Lee

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 eess.AS

Comments 34 pages, 8 figures

2006.07931 2020-08-10 eess.AS cs.DB cs.SD 83%

Solos: A Dataset for Audio-Visual Music Analysis

Juan F. Montesinos, Olga Slizovskaia, Gloria Haro

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 eess.AS

Comments Rephrased some sentenced. Explanation about OpenPose. Minor grammatical errors

2007.04687 2020-07-14 cs.CV 83%

Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision

Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, Zhiwei Yang

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV

Comments To appear in ECCV 2020

1912.00581 2020-06-17 cs.AI cs.HC q-bio.NC 83%

Optimality and limitations of audio-visual integration for cognitive systems

W. Paul Boyce, Tony Lindsay, Arkady Zgonnikov, Ignacio Rano, KongFatt Wong-Lin

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.AI

Comments 20 pages, 6 figures, 1 table 16/06/2020: Updated version includes expanded discussion and addition of new references. Also updated author affiliation information. This version has been accepted for publication with Frontiers

2005.09297 2020-05-20 eess.AS cs.LG 83%

Should we hard-code the recurrence concept or learn it instead ? Exploring the Transformer architecture for Audio-Visual Speech Recognition

George Sterpu, Christian Saam, Naomi Harte

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 eess.AS

Comments Submitted to INTERSPEECH 2020

2001.11406 2020-02-06 cs.MM cs.LG eess.IV 83%

NAViDAd: A No-Reference Audio-Visual Quality Metric Based on a Deep Autoencoder

Helard Martinez, M. C. Farias, A. Hines

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.MM

Comments 5 pages

Journal ref 2019 27th European Signal Processing Conference (EUSIPCO), IEEE, 2019, pp 1-5

2002.00163 2020-02-04 cs.CL 83%

Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog

Zekang Li, Zongjia Li, Jinchao Zhang, Yang Feng, Cheng Niu, Jie Zhou

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CL

Comments Accepted by AAAI2020 DSTC8 workshop. Ranked 1st in DSTC8-AVSD track

1909.06957 2019-09-18 cs.CV 83%

Multimodal Deep Models for Predicting Affective Responses Evoked by Movies

Ha Thi Phuong Thao, Dorien Herremans, Gemma Roig

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV

Comments 10 pages, 7 figures, Preprint accepted for publication in the Proceedings of the 2nd International Workshop on Computer Vision for Physiological Measurement as part of ICCV. Seoul, South Korea. 2019

Journal ref Proceedings of the 2nd International Workshop on Computer Vision for Physiological Measurement as part of ICCV. Seoul, South Korea. 2019

1906.10096 2019-06-25 cs.CV 83%

Audio-Visual Kinship Verification

Xiaoting Wu, Eric Granger, Xiaoyi Feng

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments 12 pages

1905.13694 2019-06-03 cs.CV 83%

Multimodal Joint Emotion and Game Context Recognition in League of Legends Livestreams

Charles Ringer, James Alfred Walker, Mihalis A. Nicolaou

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV

Comments 8 Pages, IEEE Conference on Games 2019

1809.01728 2019-05-02 eess.AS cs.LG cs.SD eess.IV stat.ML 83%

Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition

George Sterpu, Christian Saam, Naomi Harte

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 eess.AS

Comments In ICMI'18, October 16-20, 2018, Boulder, CO, USA. Equation (2) corrected on this version

1904.05876 2019-04-12 cs.CV cs.AI cs.CL cs.LG cs.SD eess.AS 83%

A Simple Baseline for Audio-Visual Scene-Aware Dialog

Idan Schwartz, Alexander Schwing, Tamir Hazan

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to CVPR 2019

1812.02872 2018-12-10 cs.CV 83%

An Attempt towards Interpretable Audio-Visual Video Captioning

Yapeng Tian, Chenxiao Guan, Justin Goodman, Marc Moore, Chenliang Xu

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments 11 pages, 4 figures

1811.05760 2018-11-15 eess.AS cs.CL cs.CV cs.MM cs.SD 83%

A Multimodal Approach towards Emotion Recognition of Music using Audio and Lyrical Content

Aniruddha Bhattacharya, K. V. Kadambari

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments 6 pages

1612.08727 2018-08-09 cs.MM cs.SD 83%

Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications

Bochen Li, Xinzhao Liu, Karthik Dinesh, Zhiyao Duan, Gaurav Sharma

专题命中音频语音多模态：multimodal(title)；multi-modal(abstract)；audio-visual(abstract)；分类 cs.MM

Comments 14 pages, 10 figures

1803.08842 2018-03-26 cs.CV 83%

Audio-Visual Event Localization in Unconstrained Videos

Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, Chenliang Xu

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments 23 pages, 7 figures

1803.04840 2018-03-14 cs.CV 83%

Resource aware design of a deep convolutional-recurrent neural network for speech recognition through audio-visual sensor fusion

Matthijs Van keirsbilck, Bert Moons, Marian Verhelst

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Tech. report

1712.09680 2018-03-02 cs.SD eess.AS 83%

A Light-Weight Multimodal Framework for Improved Environmental Audio Tagging

Juncheng Li, Yun Wang, Joseph Szurley, Florian Metze, Samarjit Das

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 eess.AS

Comments 5 pages, 3 figures, Accepted and to appear at ICASSP 2018

1711.08102 2017-12-12 cs.CV 83%

CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation

Wangli Hao, Zhaoxiang Zhang, He Guan

专题命中音频语音多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Have some problems need to be handled

1711.11017 2017-11-30 cs.AI cs.CL cs.CV cs.RO cs.SD eess.AS 83%

HoME: a Household Multimodal Environment

Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, Aaron Courville

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Presented at NIPS 2017's Visually-Grounded Interaction and Language Workshop

1706.05739 2017-11-01 cs.CV 83%

3D Convolutional Neural Networks for Cross Audio-Visual Matching Recognition

Amirsina Torfi, Seyed Mehdi Iranmanesh, Nasser M. Nasrabadi, Jeremy Dawson

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Journal ref IEEE Access (Year: 2017, Volume: PP, Issue: 99 )

1704.06913 2017-10-19 cs.CL cs.LG 83%

Learning weakly supervised multimodal phoneme embeddings

Rahma Chaabouni, Ewan Dunbar, Neil Zeghidour, Emmanuel Dupoux

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL

2312.05265 2023-12-12 cs.AI cs.CR cs.CV cs.LG eess.AS 83%

Multimodal Group Emotion Recognition In-the-wild Using Privacy-Compliant Features

Anderson Augusma, Dominique Vaufreydaz, Frédérique Letué

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、eess.AS

Journal ref ICMI '23: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION, Oct 2023, Paris, France. pp.750-754

1. 音频语音多模态 4567 篇

Disentanglement for audio-visual emotion recognition using multitask setup

Multimodal Attention Fusion for Target Speaker Extraction

Detecting expressions with multimodal transformers

Self-Supervised MultiModal Versatile Networks

Multi-modal Multi-channel Target Speech Separation

TMT: A Transformer-based Modal Translator for Improving Multimodal Sequence Representations in Audio Visual Scene-aware Dialog

SEANet: A Multi-modal Speech Enhancement Network

Correlating Subword Articulation with Lip Shapes for Embedding Aware Audio-Visual Speech Enhancement

Solos: A Dataset for Audio-Visual Music Analysis

Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision

Optimality and limitations of audio-visual integration for cognitive systems

Should we hard-code the recurrence concept or learn it instead ? Exploring the Transformer architecture for Audio-Visual Speech Recognition

NAViDAd: A No-Reference Audio-Visual Quality Metric Based on a Deep Autoencoder

Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog

Multimodal Deep Models for Predicting Affective Responses Evoked by Movies

Audio-Visual Kinship Verification

Multimodal Joint Emotion and Game Context Recognition in League of Legends Livestreams

Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition

A Simple Baseline for Audio-Visual Scene-Aware Dialog

An Attempt towards Interpretable Audio-Visual Video Captioning

A Multimodal Approach towards Emotion Recognition of Music using Audio and Lyrical Content

Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications

Audio-Visual Event Localization in Unconstrained Videos

Resource aware design of a deep convolutional-recurrent neural network for speech recognition through audio-visual sensor fusion

A Light-Weight Multimodal Framework for Improved Environmental Audio Tagging

CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation

HoME: a Household Multimodal Environment

3D Convolutional Neural Networks for Cross Audio-Visual Matching Recognition

Learning weakly supervised multimodal phoneme embeddings

Multimodal Group Emotion Recognition In-the-wild Using Privacy-Compliant Features