多模态大模型

Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Data

Yu Kang, Tianqiao Liu, Hang Li, Yang Hao, Wenbiao Ding

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments AAAI 2022

Audio-Adaptive Activity Recognition Across Video Domains

Yunhua Zhang, Hazel Doughty, Ling Shao, Cees G. M. Snoek

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments Accepted at CVPR 2022

FaceFormer: Speech-Driven 3D Facial Animation with Transformers

Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, Taku Komura

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments Accepted to CVPR 2022

Visually Supervised Speaker Detection and Localization via Microphone Array

Davide Berghi, Adrian Hilton, Philip J. B. Jackson

专题命中 音频语音多模态 ：multi-modal(abstract)；audio-visual(abstract)；分类 eess.AS

Comments Erratum: Due to a bug in the evaluation script, the correct average distance (aD) metric is here reported in yellow. The analysis remains unchanged from the original paper as the trend between the old and new measures are perfectly monotonic. The bug was caused by an incorrect normalization factor

Journal ref IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP), 2021

Novel Dual-Channel Long Short-Term Memory Compressed Capsule Networks for Emotion Recognition

Ismail Shahin, Noor Hindawi, Ali Bou Nassif, Adi Alhudhaif, Kemal Polat

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 eess.AS

Comments 19 pages, 11 figures

Journal ref Published in Expert Systems With Applications, 2021

Longitudinal Speech Biomarkers for Automated Alzheimer's Detection

Jordi Laguarta Soler, Brian Subirana

专题命中 音频语音多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 eess.AS

Journal ref Frontiers in Computer Science, 08 April 2021

Space-Time Memory Network for Sounding Object Localization in Videos

Sizhe Li, Yapeng Tian, Chenliang Xu

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments Accepted to BMVC2021. Project page: https://sites.google.com/view/bmvc2021stm

ViDA-MAN: Visual Dialog with Digital Humans

Tong Shen, Jiawei Zuo, Fan Shi, Jin Zhang, Liqin Jiang, Meng Chen, Zhengchen Zhang, Wei Zhang, Xiaodong He, Tao Mei

专题命中 音频语音多模态 ：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Drop-DTW: Aligning Common Signal Between Sequences While Dropping Outliers

Nikita Dvornik, Isma Hadji, Konstantinos G. Derpanis, Animesh Garg, Allan D. Jepson

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Exploring modality-agnostic representations for music classification

Ho-Hsiang Wu, Magdalena Fuentes, Juan P. Bello

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 eess.AS

Collaborative Learning to Generate Audio-Video Jointly

Vinod K Kurmi, Vipul Bajaj, Badri N Patro, K S Venkatesh, Vinay P Namboodiri, Preethi Jyothi

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments ICASSP 2021 (Accepted)

Beyond Image to Depth: Improving Depth Prediction using Echoes

Kranti Kumar Parida, Siddharth Srivastava, Gaurav Sharma

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments To appear in CVPR 2021

Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting

Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Yongxin Yang, Timothy M. Hospedales, Tao Xiang, Yi-Zhe Song

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021 Code : https://github.com/AyanKumarBhunia/Self-Supervised-Learning-for-Sketch

Self-supervised Contrastive Video-Speech Representation Learning for Ultrasound

Jianbo Jiao, Yifan Cai, Mohammad Alsharid, Lior Drukker, Aris T. Papageorghiou, J. Alison Noble

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments MICCAI 2020 (early acceptance)

Looking At The Body: Automatic Analysis of Body Gestures and Self-Adaptors in Psychological Distress

Weizhe Lin, Indigo Orton, Qingbiao Li, Gabriela Pavarini, Marwa Mahmoud

专题命中 音频语音多模态 ：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments 11 pages

How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition

George Sterpu, Christian Saam, Naomi Harte

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 eess.AS

Comments in IEEE/ACM Transactions on Audio, Speech, and Language Processing (to appear)

Exploring Context, Attention and Audio Features for Audio Visual Scene-Aware Dialog

Shachi H Kumar, Eda Okur, Saurav Sahay, Jonathan Huang, Lama Nachman

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CL

Comments Presented at the Visual Question Answering and Dialog Workshop, CVPR 2019, Long Beach, USA. arXiv admin note: substantial text overlap with arXiv:1912.10131

Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision

Aren Jansen, Daniel P. W. Ellis, Shawn Hershey, R. Channing Moore, Manoj Plakal, Ashok C. Popat, Rif A. Saurous

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments This extended version of a ICASSP 2020 submission under same title has an added figure and additional discussion for easier consumption

AV Speech Enhancement Challenge using a Real Noisy Corpus

Mandar Gogate, Ahsan Adeel, Kia Dashtipour, Peter Derleth, Amir Hussain

专题命中 音频语音多模态 ：multi-modal(abstract)；audio-visual(abstract)；分类 eess.AS

Comments arXiv admin note: substantial text overlap with arXiv:1909.10407

Time Domain Audio Visual Speech Separation

Jian Wu, Yong Xu, Shi-Xiong Zhang, Lian-Wu Chen, Meng Yu, Lei Xie, Dong Yu

专题命中 音频语音多模态 ：multi-modal(abstract)；audio-visual(abstract)；分类 eess.AS

Comments Accepted to ASRU 2019

2.5D Visual Sound

Ruohan Gao, Kristen Grauman

专题命中 音频语音多模态 ：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments Published in CVPR 2019, project page: http://vision.cs.utexas.edu/projects/2.5D_visual_sound/

Context, Attention and Audio Feature Explorations for Audio Visual Scene-Aware Dialog

Shachi H Kumar, Eda Okur, Saurav Sahay, Juan Jose Alvarado Leanos, Jonathan Huang, Lama Nachman

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CL

Comments 7 pages, 2 figures, DSTC7 workshop at AAAI 2019

Visual Features for Context-Aware Speech Recognition

Abhinav Gupta, Yajie Miao, Leonardo Neves, Florian Metze

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 5 pages and 3 figures

Journal ref IEEE Xplore (ICASSP) (2017) 5020-5024

Robust end-to-end deep audiovisual speech recognition

Ramon Sanabria, Florian Metze, Fernando De La Torre

专题命中 音频语音多模态 ：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CL

MarginNCE: Robust Sound Localization with a Negative Margin

Sooyoung Park, Arda Senocak, Joon Son Chung

专题命中 音频语音多模态 ：audio-visual(abstract,comments)；分类 cs.CV、cs.MM、eess.AS

Comments Submitted to ICASSP 2023. SOTA performance in Audio-Visual Sound Localization. 5 Pages

What is Multimodality?

Letitia Parcalabescu, Nils Trost, Anette Frank

专题命中 音频语音多模态 ：multimodal(abstract,journal_ref)；分类 cs.CV、cs.CL、cs.AI

Comments Paper accepted for publication at MMSR 2021; 10 pages, 5 figures

Journal ref Proceedings of the 1st Workshop on Multimodal Semantic Representations (MMSR), 2021, Groningen, Netherlands (Online), Association for Computational Linguistics, p. 1--10

When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue

当文本误导时：面向音频接地对话的不一致感知推理

Yen-Ju Lu, Yuzhe Wang, Yaohan Guan, Xiluo He, Jiarui Hai, Mingrui Liang, Kaavya Chaparala, Thomas Thebaud, Laureano Moro-Velazquez, Najim Dehak, Jesus Villalba

机构 * Center for Language and Speech Processing, Johns Hopkins University（约翰霍普金斯大学语言与语音处理中心） 纠错

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

AI总结 本研究针对口语对话理解中基于转录本的捷径问题，构建了含501个问题的受控基准ContraTalk，提出Audio Twin智能体式推理框架，可提升冲突问答案例的准确率并减少文本偏向陷阱选择。

Comments 24 pages, 4 figures

ONOTE: Hypergraph-Grounded Omnimodal Reasoning for Computational Music Science

ONOTE：面向专家级音乐智能的多模态记谱处理基准测试

Menghe Ma, Siqing Wei, Yuecheng Xing, Ziyue Zhu, Zhenghong Lin, Yaheng Wang, Fanhong Meng, Peijun Han, Luu Anh Tuan, Haoran Luo

机构 * Beijing University of Posts and Telecommunications（北京邮电大学） ； Nanyang Technological University（南洋理工大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

AI总结 本文提出ONOTE基准测试，通过确定性流程消除记谱系统偏差，揭示多模态模型在感知准确性和音乐理论理解间的根本分歧。

MUSE: A Heterogeneity-Aware Multimedia Search Engine for Mobile SoCs

AME：一种高效的异构代理记忆引擎用于智能手机

Xinkui Zhao, Qingyu Ma, Yifan Zhang, Hengxuan Lou, Sai Liu, Chang Liu, Guanjie Cheng, Naibo Wang, Yueshen Xu

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)

AI总结 AME是一种为智能手机设计的高效异构代理记忆引擎，通过硬件感知的矩阵流水线和调度方案提升内存处理效率，实现更高吞吐量和更低延迟。

Comments Accepted by the 34th ACM International Conference on Multimedia (MM '26). 9 pages, 11 figures

A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook

大型音频语言模型综述：通用性、可信度与展望

Kaiwen Luo, Zhenhong Zhou, Leyan Wang, Liang Lin, Tianyu Shao, Yuanhe Zhang, Yang Xiao, Yuxuan Li, Miao Yu, Kailin Lyu, Jiaming Zhang, Li Sun, Songze Li, Yueming Wu, Ting Dang, Xiaojun Jia, Dongrui Liu, Kai Li, Rohan Kumar Das, Siyuan Liang, Xinfeng Li, Qiankun Li, Jing Chen, Xingjun Ma, Kun Wang, Junhao Dong, Deqing Zou, Yu Cheng, Xia Hu, Zhigang Zeng, Sen Su, Yang Liu, Yu-Gang Jiang, Philip S. Yu, Yew-Soon Ong

机构 * Nanyang Technological University（南洋理工大学） ； Independent Researcher（独立研究者） ； The University of Melbourne（墨尔本大学） ； North China Electric Power University（华北电力大学） ； Beijing University of Posts and Telecommunications（北京邮电大学） ； University of Chinese Academy of Sciences（中国科学院大学） ； University of Science and Technology of China（中国科学技术大学） ； Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所） ； Shanghai AI Laboratory（上海人工智能实验室） ； Huazhong University of Science and Technology（华中科技大学） ； Tsinghua University（清华大学） ； Fortemedia Singapore（富媒体新加坡） ； Tencent（腾讯） ； Fudan University（复旦大学） ； Wuhan University（武汉大学） ； Chinese University of Hong Kong（香港中文大学） ； Chongqing University of Posts and Telecommunications（重庆邮电大学） ； University of Illinois Chicago（伊利诺伊大学芝加哥分校） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)

AI总结 本文综述了大型音频语言模型的通用性、可信度及未来发展方向，探讨了其架构创新、对齐算法及安全风险，并提出了防御深入、因果音频世界建模等策略以提升音频智能的可信度。