多模态大模型

Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding

Zengjie Song, Zhaoxiang Zhang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to IEEE Transactions on Neural Networks and Learning Systems (T-NNLS)

MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition

Yuchen Hu, Chen Chen, Ruizhe Li, Heqing Zou, Eng Siong Chng

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 14 pages, 5 figures, Accepted by ACL 2023

Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention

Xubo Liu, Qiushi Huang, Xinhao Mei, Haohe Liu, Qiuqiang Kong, Jianyuan Sun, Shengchen Li, Tom Ko, Yu Zhang, Lilian H. Tang, Mark D. Plumbley, Volkan Kılıç, Wenwu Wang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments INTERSPEECH 2023

AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation

Shentong Mo, Yapeng Tian

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline

Tiantian Geng, Teng Wang, Jinming Duan, Runmin Cong, Feng Zheng

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by CVPR2023

Egocentric Audio-Visual Object Localization

Chao Huang, Yapeng Tian, Anurag Kumar, Chenliang Xu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by CVPR 2023

Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring

Joanna Hong, Minsu Kim, Jeongsoo Choi, Yong Man Ro

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted at CVPR 2023. Implementation available: https://github.com/joannahong/AV-RelScore

Speaker Recognition in Realistic Scenario Using Multimodal Data

Saqlain Hussain Shah, Muhammad Saad Saeed, Shah Nawaz, Muhammad Haroon Yousaf

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted at the International Conference on Artificial Intelligence (ICAI'2023)

Cross-Modal Mutual Learning for Cued Speech Recognition

Lei Liu, Li Liu

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Accepted to ICASSP2023

AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments

Sudipta Paul, Amit K. Roy-Chowdhury, Anoop Cherian

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at NeurIPS 2022

Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectors

Vladimir Iashin, Weidi Xie, Esa Rahtu, Andrew Zisserman

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted as a spotlight presentation for the BMVC 2022. Code: https://github.com/v-iashin/SparseSync Project page: https://v-iashin.github.io/SparseSync

Audio-Visual Scene Classification Using A Transfer Learning Based Joint Optimization Strategy

Chengxin Chen, Meng Wang, Pengyuan Zhang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 5 pages, 2 figures, based on the work that won first place in the challenge of DCASE2021 Task 1B

A Study of Designing Compact Audio-Visual Wake Word Spotting System Based on Iterative Fine-Tuning in Neural Network Pruning

Hengshun Zhou, Jun Du, Chao-Han Huck Yang, Shifu Xiong, Chin-Hui Lee

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Accepted to ICASSP 2022. H. Zhou et al

LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences

Ziwang Fu, Feng Liu, Hanyang Wang, Siyuan Shen, Jiahao Zhang, Jiayin Qi, Xiangling Fu, Aimin Zhou

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments 9 pages ,Figure 2, Table 5

Squeeze-Excitation Convolutional Recurrent Neural Networks for Audio-Visual Scene Classification

Javier Naranjo-Alcazar, Sergi Perez-Castanos, Aaron Lopez-Garcia, Pedro Zuccarello, Maximo Cobos, Francesc J. Ferri

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Exploring Deep Learning for Joint Audio-Visual Lip Biometrics

Meng Liu, Longbiao Wang, Kong Aik Lee, Hanyi Zhang, Chang Zeng, Jianwu Dang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Cross-modal Speaker Verification and Recognition: A Multilingual Perspective

Muhammad Saad Saeed, Shah Nawaz, Pietro Morerio, Arif Mahmood, Ignazio Gallo, Muhammad Haroon Yousaf, Alessio Del Bue

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Accepted: CVPRW

FaceFilter: Audio-visual speech separation using still images

Soo-Whan Chung, Soyeon Choe, Joon Son Chung, Hong-Goo Kang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Under submission as a conference paper. Video examples: https://youtu.be/ku9xoLh62E

Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning

Ying Cheng, Ruize Wang, Zhihao Pan, Rui Feng, Yuejie Zhang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by the 28th ACM International Conference on Multimedia (ACM MM 2020)

Multi-Modal Music Information Retrieval: Augmenting Audio-Analysis with Visual Computing for Improved Music Video Analysis

Alexander Schindler

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Dissertation at TU Wien

Leveraging Topics and Audio Features with Multimodal Attention for Audio Visual Scene-Aware Dialog

Shachi H Kumar, Eda Okur, Saurav Sahay, Jonathan Huang, Lama Nachman

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Presented at the 3rd Visually Grounded Interaction and Language (ViGIL) Workshop, NeurIPS 2019, Vancouver, Canada. arXiv admin note: substantial text overlap with arXiv:1812.08407, arXiv:1912.10132

Modality Attention for End-to-End Audio-visual Speech Recognition

Pan Zhou, Wenwen Yang, Wei Chen, Yanfeng Wang, Jia Jia

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments accepted by ICASSP2019

VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model

Zuwei Long, Yunhang Shen, Chaoyou Fu, Heting Gao, Lijiang Li, Peixian Chen, Mengdan Zhang, Hang Shao, Jian Li, Jinlong Peng, Haoyu Cao, Ke Li, Rongrong Ji, Xing Sun

机构 * Tencent Youtu Lab（腾讯优图实验室） ； Nanjing University（南京大学） ； Xiamen University（厦门大学） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CL、cs.AI；MLLM(comments)

Comments Training and Inference Codes: https://github.com/VITA-MLLM/VITA-Audio

MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks

Sanjoy Chowdhury, Mohamed Elmoghany, Yohan Abeysinghe, Junjie Fei, Sayan Nag, Salman Khan, Mohamed Elhoseiny, Dinesh Manocha

机构 * University of Maryland, College Park（马里兰大学College Park分校） ； KAUST（卡尔斯兰大学） ； MBZUAI（马克斯·普朗克人工智能研究所） ； University of Toronto（多伦多大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments Audio-visual learning, Audio-Visual RAG, Multi-Video Linkage

Learning in Audio-visual Context: A Review, Analysis, and New Perspective

Yake Wei, Di Hu, Yapeng Tian, Xuelong Li

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments https://gewu-lab.github.io/audio-visual-learning/

Semantic Audio-Visual Navigation

Changan Chen, Ziad Al-Halah, Kristen Grauman

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、eess.AS

Comments Project page: http://vision.cs.utexas.edu/projects/semantic-audio-visual-navigation

AV-EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Omni-modal LLMS with Audio-visual Cues

AV-EMO-Reasoning: 在具有视听线索的全模态大语言模型中基准测试情感推理能力

Dingkun Zhou, Krish Patel, Ajay Kankipati, Akshaj Gupta, Zeyi Austin Li, Mohul Shukla, Vibhor Narang, Sara Kofman, Zongli Ye, Grace Wang, Xiaoyu Shi, Tingle Li, Guan-Ting Lin, Kan Jen Cheng, Huang-Cheng Chou, Jiachen Lian, Gopala Anumanchipalli

机构 * UC Berkeley（加州大学伯克利分校） ； South China University of Technology（华南理工大学） ； Zhejiang University（浙江大学） ； National Taiwan University（台湾大学） ； University of Southern California（美国南加州大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title)；omni-modal(title)；分类 cs.MM

AI总结 提出AV-EMO-Reasoning基准，通过合成和真实世界的视听对话数据集及情感感知与交互推理指标，系统评估全模态大语言模型的情感推理能力。

Valley3: Scaling Omni Foundation Models for E-commerce

Valley3：面向电商的多模态大语言模型规模化

Zeyu Chen, Guanghao Zhou, Qixiang Yin, Ziwang Zhao, Huanjin Yao, Pengjiu Xia, Min Yang, Cen Chen, Minghui Qiu

机构 * Valley Team, ByteDance Group（字节跳动集团山谷团队） 纠错

专题命中 音频语音多模态 ：MLLM(abstract,abstract_cn)；multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)

AI总结 Valley3通过四阶段预训练管道，融合多模态能力与电商知识，实现跨模态指令跟随和长链推理，构建了涵盖6个任务的电商基准测试，展现出在电商场景中的优越性能。

ATIR: Towards Audio-Text Interleaved Contextual Retrieval

ATIR：面向音频-文本交错上下文检索

Tong Zhao, Chenghao Zhang, Yutao Zhu, Zhicheng Dou

机构 * Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学北京校区人工智能学院） 纠错

专题命中 音频语音多模态 ：MLLM(summary_cn,abstract)；multimodal(abstract)；分类 cs.AI

AI总结 本文提出ATIR任务，通过整合ASR、QA和检索数据集构建基准，解决现有音频检索数据集在语义检索上的局限，引入新的token压缩机制提升MLLM在ATIR中的性能。

HAVT-IVD: Heterogeneity-Aware Cross-Modal Network for Audio-Visual Surveillance: Idling Vehicles Detection With Multichannel Audio and Multiscale Visual Cues

Xiwen Li, Xiaoya Tang, Tolga Tasdizen

机构 * Scientific Computing and Imaging Institute（科学计算与成像研究所） ； Department of Electrical and Computer Engineering（电气与计算机工程系） ； University of Utah（犹他大学） 纠错

专题命中 音频语音多模态 ：cross-modal(title)；audio-visual(title)；分类 cs.CV