多模态大模型 - arXivDaily 专题

2408.06753 2024-10-15 cs.CV cs.MM cs.SD eess.AS 82%

Detecting Audio-Visual Deepfakes with Fine-Grained Inconsistencies

Marcella Astrid, Enjie Ghorbel, Djamila Aouada

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted in BMVC 2024

2409.19575 2024-10-01 cs.SD cs.CL cs.MM eess.AS 82%

Quantitative Analysis of Audio-Visual Tasks: An Information-Theoretic Perspective

Chen Chen, Xiaolou Li, Zehua Liu, Lantian Li, Dong Wang

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted by ISCSLP2024

2409.19132 2024-10-01 cs.MM cs.CV cs.LG cs.SD eess.AS 82%

From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation

Kun Su, Xiulong Liu, Eli Shlizerman

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by ICML 2024

2409.18971 2024-10-01 cs.MM cs.AI cs.SD eess.AS 82%

Early Joint Learning of Emotion Information Makes MultiModal Model Understand You Better

Mengying Ge, Mingyang Li, Dongkai Tang, Pengbo Li, Kuo Liu, Shuhao Deng, Songbai Pu, Long Liu, Yang Song, Tao Zhang

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.AI、cs.MM、eess.AS

2406.18088 2024-09-25 cs.CL cs.AI cs.SD eess.AS 82%

LLM-Driven Multimodal Opinion Expression Identification

Bonian Jia, Huiyao Chen, Yueheng Sun, Meishan Zhang, Min Zhang

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 5 pages, 3 Figures, Accept by Interspeech 2024

Journal ref Proceedings of Interspeech 2024

2409.14340 2024-09-24 cs.CV cs.LG cs.MM cs.SD eess.AS 82%

Self-Supervised Audio-Visual Soundscape Stylization

Tingle Li, Renhao Wang, Po-Yao Huang, Andrew Owens, Gopala Anumanchipalli

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments ECCV 2024

2409.11729 2024-09-19 cs.MM cs.CV cs.SD eess.AS 82%

DETECLAP: Enhancing Audio-Visual Representation Learning with Object Information

Shota Nakada, Taichi Nishimura, Hokuto Munakata, Masayoshi Kondo, Tatsuya Komatsu

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments under review

2409.09611 2024-09-17 cs.CV cs.AI cs.LG cs.SD eess.AS 82%

Integrating Audio Narrations to Strengthen Domain Generalization in Multimodal First-Person Action Recognition

Cagri Gungor, Adriana Kovashka

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、eess.AS

2409.06709 2024-09-12 cs.MM cs.AI cs.SD eess.AS 82%

Unveiling Visual Biases in Audio-Visual Localization Benchmarks

Liangyu Chen, Zihao Yue, Boshen Xu, Qin Jin

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.AI、cs.MM、eess.AS

Comments Accepted by ECCV24 AVGenL Workshop

2409.02239 2024-09-06 cs.SD cs.AI cs.CL eess.AS 82%

Temporal Order Preserved Optimal Transport-based Cross-modal Knowledge Transfer Learning for ASR

Xugang Lu, Peng Shen, Yu Tsao, Hisashi Kawai

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted to IEEE SLT 2024

2406.14176 2024-08-20 cs.SD cs.AI cs.MM eess.AS 82%

A Multi-Stream Fusion Approach with One-Class Learning for Audio-Visual Deepfake Detection

Kyungbok Lee, You Zhang, Zhiyao Duan

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.AI、cs.MM、eess.AS

2408.01337 2024-08-05 cs.SD cs.CL cs.LG cs.MM eess.AS 82%

MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models

Benno Weck, Ilaria Manco, Emmanouil Benetos, Elio Quinton, George Fazekas, Dmitry Bogdanov

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted at ISMIR 2024. Data: https://doi.org/10.5281/zenodo.12709974 Code: https://github.com/mulab-mir/muchomusic Supplementary material: https://mulab-mir.github.io/muchomusic

2303.17395 2024-07-22 eess.AS cs.CL cs.MM cs.SD 82%

WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research

Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, Wenwu Wang

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted to TASLP

2407.11650 2024-07-18 cs.CV cs.MM cs.SD eess.AS 82%

Statistics-aware Audio-visual Deepfake Detector

Marcella Astrid, Enjie Ghorbel, Djamila Aouada

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted in ICIP 2024

2407.11492 2024-07-17 cs.SD cs.CL cs.MM eess.AS 82%

MMSD-Net: Towards Multi-modal Stuttering Detection

Liangyu Nie, Sudarsana Reddy Kadiri, Ruchit Agrawal

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted at INTERSPEECH 2024

2407.08130 2024-07-12 cs.MM cs.CV cs.SD eess.AS 82%

Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning

Wenrui Li, Penghong Wang, Ruiqin Xiong, Xiaopeng Fan

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by TIP

2407.07825 2024-07-11 cs.SD cs.CV cs.MM eess.AS 82%

RT-LA-VocE: Real-Time Low-SNR Audio-Visual Speech Enhancement

Honglie Chen, Rodrigo Mira, Stavros Petridis, Maja Pantic

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Interspeech 2024

2407.02004 2024-07-08 cs.CV cs.AI cs.SD eess.AS 82%

SAVE: Segment Audio-Visual Easy way using Segment Anything Model

Khanh-Binh Nguyen, Chae Jung Park

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.AI、eess.AS

2406.15754 2024-06-25 cs.CV cs.CL cs.LG cs.SD eess.AS 82%

Multimodal Segmentation for Vocal Tract Modeling

Rishi Jain, Bohan Yu, Peter Wu, Tejas Prabhune, Gopala Anumanchipalli

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Interspeech 2024

2406.13384 2024-06-21 cs.SD cs.CV cs.MM eess.AS 82%

Straight Through Gumbel Softmax Estimator based Bimodal Neural Architecture Search for Audio-Visual Deepfake Detection

Aravinda Reddy PN, Raghavendra Ramachandra, Krothapalli Sreenivasa Rao, Pabitra Mitra, Vinod Rathod

专题命中音频语音多模态：audio-visual(title)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

2406.09618 2024-06-17 cs.CL cs.AI cs.IR cs.SD eess.AS 82%

Multi-Modal Retrieval For Large Language Model Based Speech Recognition

Jari Kolehmainen, Aditya Gourav, Prashanth Gurunath Shivakumar, Yile Gu, Ankur Gandhe, Ariya Rastrow, Grant Strimel, Ivan Bulyko

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

2406.09569 2024-06-17 cs.CL cs.AI cs.SD eess.AS 82%

Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Time

Frank Seide, Morrie Doulaty, Yangyang Shi, Yashesh Gaur, Junteng Jia, Chunyang Wu

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

2406.09156 2024-06-14 cs.LG cs.CV cs.MM cs.SD eess.AS 82%

Towards Multilingual Audio-Visual Question Answering

Orchid Chetia Phukan, Priyabrata Mallick, Swarup Ranjan Behera, Aalekhya Satya Narayani, Arun Balaji Buduru, Rajesh Sharma

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to Interspeech 2024

2406.06438 2024-06-11 cs.CL cs.CV cs.HC cs.LG cs.SD eess.AS 82%

Multimodal Contextualized Semantic Parsing from Speech

Jordan Voas, Raymond Mooney, David Harwath

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments 10 Pages, 3 figures, ACL 2024 Main

2406.01321 2024-06-04 cs.SD cs.AI cs.LG cs.MM eess.AS 82%

Sequence-to-Sequence Multi-Modal Speech In-Painting

Mahsa Kadkhodaei Elyaderani, Shahram Shirani

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.AI、cs.MM、eess.AS

2406.00901 2024-06-04 cs.MM cs.AI cs.LG cs.SD eess.AS 82%

Robust Multi-Modal Speech In-Painting: A Sequence-to-Sequence Approach

Mahsa Kadkhodaei Elyaderani, Shahram Shirani

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.AI、cs.MM、eess.AS

2401.13260 2024-05-29 cs.CL cs.MM cs.SD eess.AS 82%

MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion, Asr Error Detection, and Asr Error Correction

Jiajun He, Xiaohan Shi, Xingfeng Li, Tomoki Toda

专题命中音频语音多模态：multimodal(title)；multi-modal(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted by ICASSP 2024

2404.07336 2024-04-12 cs.CV cs.MM eess.AS 82%

PEAVS: Perceptual Evaluation of Audio-Visual Synchrony Grounded in Viewers' Opinion Scores

Lucas Goncalves, Prashant Mathur, Chandrashekhar Lavania, Metehan Cekic, Marcello Federico, Kyu J. Han

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 24 pages

2404.00511 2024-04-12 cs.CL cs.CV cs.MM 82%

MIPS at SemEval-2024 Task 3: Multimodal Emotion-Cause Pair Extraction in Conversations with Multimodal Language Models

Zebang Cheng, Fuqiang Niu, Yuxiang Lin, Zhi-Qi Cheng, Bowen Zhang, Xiaojiang Peng

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Ranked 3rd in SemEval '24 Task 3 with F1 of 0.3435, close to 1st & 2nd by 0.0339 & 0.0025

2312.06462 2024-04-09 cs.CV cs.AI cs.SD eess.AS 82%

Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation

Qi Yang, Xing Nie, Tong Li, Pengfei Gao, Ying Guo, Cheng Zhen, Pengfei Yan, Shiming Xiang

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.AI、eess.AS

Comments CVPR 2024 Highlight. 13 pages, 10 figures

1. 音频语音多模态 4578 篇

Detecting Audio-Visual Deepfakes with Fine-Grained Inconsistencies

Quantitative Analysis of Audio-Visual Tasks: An Information-Theoretic Perspective

From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation

Early Joint Learning of Emotion Information Makes MultiModal Model Understand You Better

LLM-Driven Multimodal Opinion Expression Identification

Self-Supervised Audio-Visual Soundscape Stylization

DETECLAP: Enhancing Audio-Visual Representation Learning with Object Information

Integrating Audio Narrations to Strengthen Domain Generalization in Multimodal First-Person Action Recognition

Unveiling Visual Biases in Audio-Visual Localization Benchmarks

Temporal Order Preserved Optimal Transport-based Cross-modal Knowledge Transfer Learning for ASR

A Multi-Stream Fusion Approach with One-Class Learning for Audio-Visual Deepfake Detection

MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models

WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research

Statistics-aware Audio-visual Deepfake Detector

MMSD-Net: Towards Multi-modal Stuttering Detection

Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning

RT-LA-VocE: Real-Time Low-SNR Audio-Visual Speech Enhancement

SAVE: Segment Audio-Visual Easy way using Segment Anything Model

Multimodal Segmentation for Vocal Tract Modeling

Straight Through Gumbel Softmax Estimator based Bimodal Neural Architecture Search for Audio-Visual Deepfake Detection

Multi-Modal Retrieval For Large Language Model Based Speech Recognition

Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Time

Towards Multilingual Audio-Visual Question Answering

Multimodal Contextualized Semantic Parsing from Speech

Sequence-to-Sequence Multi-Modal Speech In-Painting

Robust Multi-Modal Speech In-Painting: A Sequence-to-Sequence Approach

MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion, Asr Error Detection, and Asr Error Correction

PEAVS: Perceptual Evaluation of Audio-Visual Synchrony Grounded in Viewers' Opinion Scores

MIPS at SemEval-2024 Task 3: Multimodal Emotion-Cause Pair Extraction in Conversations with Multimodal Language Models

Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation