多模态大模型

On Adversarial Robustness of Large-scale Audio Visual Learning

Juncheng B Li, Shuhui Qu, Xinjian Li, Po-Yao Huang, Florian Metze

专题命中 音频语音多模态 ：multi-modal(abstract)；audio-visual(abstract)；分类 cs.MM、eess.AS

Journal ref 2022 International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2022)

Multimedia Datasets for Anomaly Detection: A Review

Pratibha Kumari, Anterpreet Kaur Bedi, Mukesh Saini

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM

Comments 17 pages, 11 figures, 8 tables

Continuous Emotion Recognition using Visual-audio-linguistic information: A Technical Report for ABAW3

Su Zhang, Ruyi An, Yi Ding, Cuntai Guan

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments 5 pages, 1 figure. arXiv admin note: substantial text overlap with arXiv:2107.01175

Facial Expression Recognition with Swin Transformer

Jun-Hwa Kim, Namho Kim, Chee Sun Won

专题命中 音频语音多模态 ：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI

STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation

Qingkai Fang, Rong Ye, Lei Li, Yang Feng, Mingxuan Wang

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI

Comments ACL 2022 main conference

Binaural SoundNet: Predicting Semantics, Depth and Motion with Binaural Sounds

Dengxin Dai, Arun Balajee Vasudevan, Jiri Matas, Luc Van Gool

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted by TPAMI. arXiv admin note: substantial text overlap with arXiv:2003.04210

Audio Visual Scene-Aware Dialog Generation with Transformer-based Video Representations

Yoshihiro Yamazaki, Shota Orihashi, Ryo Masumura, Mihiro Uchida, Akihiko Takashima

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL

Comments Accepted at DSTC10 Workshop at AAAI 2022

Learning music audio representations via weak language supervision

Ilaria Manco, Emmanouil Benetos, Elio Quinton, Gyorgy Fazekas

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Comments Accepted to ICASSP 2022

Sound-Guided Semantic Image Manipulation

Seung Hyun Lee, Wonseok Roh, Wonmin Byeon, Sang Ho Yoon, Chan Young Kim, Jinkyu Kim, Sangpil Kim

专题命中 音频语音多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、eess.AS

TEASEL: A Transformer-Based Speech-Prefixed Language Model

Mehdi Arjmand, Mohammad Javad Dousti, Hadi Moradi

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI

AI Choreographer: Music Conditioned 3D Dance Generation with AIST++

Ruilong Li, Shan Yang, David A. Ross, Angjoo Kanazawa

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Project page: https://google.github.io/aichoreographer/; Dataset page: https://google.github.io/aistplusplus_dataset/

MusicTM-Dataset for Joint Representation Learning among Sheet Music, Lyrics, and Musical Audio

Donghuo Zeng, Yi Yu, Keizo Oyama

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.MM、eess.AS

Comments 12 pages, 5 figures, 2 tables

Journal ref CSMT2020

Sound2Sight: Generating Visual Dynamics from Sound and Context

Anoop Cherian, Moitreya Chatterjee, Narendra Ahuja

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted at ECCV 2020

From Inference to Generation: End-to-end Fully Self-supervised Generation of Human Face from Speech

Hyeong-Seok Choi, Changdae Park, Kyogu Lee

专题命中 音频语音多模态 ：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments 18 pages, 12 figures, Published as a conference paper at International Conference on Learning Representations (ICLR) 2020. (camera-ready version)

Self-supervised Moving Vehicle Tracking with Stereo Sound

Chuang Gan, Hang Zhao, Peihao Chen, David Cox, Antonio Torralba

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments To appear at ICCV 2019. Project page: http://sound-track.csail.mit.edu

LRS3-TED: a large-scale dataset for visual speech recognition

Triantafyllos Afouras, Joon Son Chung, Andrew Zisserman

专题命中 音频语音多模态 ：audio-visual(abstract,comments)；multi-modal(abstract)；分类 cs.CV

Comments The audio-visual dataset can be downloaded from http://www.robots.ox.ac.uk/~vgg/data/lip_reading/

Multimodal-Language-Model-Driven Interaction and Companionship for Service Robots in Elderly-Care Facilities

面向养老机构服务机器人的多模态语言模型驱动的交互与陪伴

Ching-Chieh Liu, Cong-Thanh Vu, Yen-Chen Liu

机构 * National Cheng Kung University (NCKU)（成功大学） 纠错

专题命中 音频语音多模态 ：multimodal(title)

AI总结 该研究针对养老机构服务机器人缺乏综合陪伴与安全能力的问题，提出整合主动视觉跟人、LLM语音交互及VLM安全监测的智能陪伴机器人系统，实验验证其跟随交互与跌倒检测的有效性。

Comments Accepted to the 2026 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM)

omni-macos: On-Device Omni-Modal Search on Apple Silicon

omni-macos：在Apple Silicon上运行的端侧全模态搜索系统

Han Xiao

专题命中 音频语音多模态 ：omni-modal(title)

AI总结 omni-macos是一款在Apple Silicon端侧运行的全模态搜索系统，可在用户Mac设备上处理多模态数据搜索，数据不离开设备，适配不同硬件规格的Mac并优化内存使用。

Comments 18 pages, 5 figures, 10 tables

PI -- Multimodal Planetary Defense

PI -- 多模态行星防御

Philip Lubin

专题命中 音频语音多模态 ：multimodal(title)

AI总结 本文提出了一种高效的行星防御方法，通过高速动能穿甲弹粉碎小行星，可在不同预警时间内有效防御直径20-1000米的天体。

Comments 195 pages, 148 figures. Published in Advances in Space Research (ASR) 10-22; https://www.sciencedirect.com/science/article/pii/S0273117722009395

SocialPulse: On-Device Detection of Social Interactions in Naturalistic Settings Using Smartwatch Sensing

SocialPulse: 使用智能手表多模态感知在自然环境中检测社交互动

Md Sabbir Ahmed, Kaitlyn Dorothy Petz, Noah French, Tanvi Lakhtakia, Aayushi Sangani, Mark Rucker, Xinyu Chen, Bethany A. Teachman, Laura E. Barnes

专题命中 音频语音多模态 ：multimodal(title)

AI总结 本文提出了一种在设备上检测自然环境中社交互动的方法，通过智能手表的多模态感知技术，实现了对多种社交互动的准确识别，并在真实场景中验证了其有效性。

Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings

赋予传感器声音：用于语义时间序列嵌入的多模态JEPA

Utsav Dutta, Gerardo Pastrana, Sina Khoshfetrat Pakazad, Henrik Ohlsson

机构 * C3 AI 纠错

专题命中 音频语音多模态 ：multimodal(title)

AI总结 提出CHARM模型，通过通道级文本描述与Transformer编码器结合，利用联合嵌入预测架构（JEPA）学习语义时间序列嵌入，在异常检测、分类和预测任务中仅用线性探针即取得强性能。

Comments 9 pages, 5 figures, accepted at ICML 2026. arXiv admin note: substantial text overlap with arXiv:2505.14543

Journal ref Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 2026

GuardReasoner-Omni: A Reasoning-based Multi-modal Guardrail for Text, Image, Video, and Audio

GuardReasoner-Omni：一种基于推理的多模态护栏，适用于文本、图像、视频和音频

Zhenhao Zhu, Yue Liu, Yanpei Guo, Wenjie Qu, Cancan Chen, Yufei He, Yibo Li, Yulin Chen, Tianyi Wu, Huiying Xu, Xinzhong Zhu, Jiaheng Zhang

专题命中 音频语音多模态 ：multi-modal(title)

AI总结 提出GuardReasoner-Omni，一种基于推理的多模态护栏模型，通过两阶段训练（SFT和RL）在3B和7B参数规模上实现对文本、图像、视频和音频数据的有效审核，性能优于现有基线。

Gesture First, LLM-Assisted Voice Complement: Exploring Multimodal Robot 'Puppeteer' Teleoperation Via Virtual Counterpart in Augmented Reality

先手势，LLM辅助语音补充：通过增强现实中的虚拟对应物探索多模态机器人'提线人'遥控

Yuchong Zhang, Bastian Orthmann, Shichen Ji, Michael Welle, Jonne Van Haastregt, Danica Kragic

机构 * KTH Royal Institute of Technology（皇家理工学院） 纠错

专题命中 音频语音多模态 ：multimodal(title)

AI总结 本文探讨了通过增强现实中的虚拟对应物实现多模态机器人遥控的方法，比较了仅手势和结合语音与手势的交互方式在性能和用户体验上的差异，提出设计指南以平衡效率、鲁棒性和用户专业性。

Comments This work is under peer review

Feelings, Not Feel: Affective Audio-Visual Pseudo-Haptics in Hand-Tracked XR

感受，而非感受：手部追踪XR中的情感音频视觉伪触觉

Kristian Paolo David, Tyrone Justin Sta Maria, Mikkel Dominic Gamboa, Jordan Aiko Deja

专题命中 音频语音多模态 ：audio-visual(title)

AI总结 该研究探讨了在无控制器的XR环境中，通过伪触觉线索（音频、视觉）影响用户情感体验的机制，发现其更应视为情感反馈而非直接触觉替代。

Comments 5 pages, 2 figures, 1 table, CHI EA Posters

Journal ref Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems (CHI EA '26), April 13--17, 2026, Barcelona, Spain

Dynamite: Real-Time Debriefing Slide Authoring through AI-Enhanced Multimodal Interaction

Dynamite：通过AI增强的多模态交互实现实时复盘幻灯片创作

Panayu Keelawat, David Barron, Kaushik Narasimhan, Daniel Manesh, Xiaohang Tang, Xi Chen, Sang Won Lee, Yan Chen

专题命中 音频语音多模态 ：multimodal(title)

AI总结 Dynamite通过AI增强的多模态交互实现实时复盘幻灯片创作，提升课堂讨论后复盘的准确性和效率。

Comments Accepted to VL/HCC 2025

Generative Multi-modal Feedback for Singing Voice Synthesis Evaluation

生成式多模态反馈用于歌唱语音合成评估

Xueyan Li, Yuxin Wang, Mengjie Jiang, Qingzi Zhu, Jiang Zhang, Zoey Kim, Yazhe Niu

机构 * Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） ； University of Science and Technology of China（中国科学技术大学） ； Columbia University（哥伦比亚大学） ； Dalian University of Technology（大连理工大学） ； The Chinese University of Hong Kong（香港中文大学） 纠错

专题命中 音频语音多模态 ：multi-modal(title)

AI总结 本文提出生成式多模态反馈框架，通过音频-语言模型生成多维语言和音频反馈，提升歌唱语音合成评估的准确性和可解释性。

Comments 16 pages, 5 figures

Colin: A Multimodal Human-AI Co-Creation Storytelling System To Support Children's Multi-Level Narrative Skills

Colin：一种多模态人机协同创故事叙述系统，以支持儿童的多层级叙述技能

Lyumanshan Ye, Jiandong Jiang, Yuhan Liu, Yihan Ran, Yufan Zhou, Zhao Wang, Yipeng Yu, Pengfei Liu, Danni Chang, Yucheng Jin

专题命中 音频语音多模态 ：multimodal(title)

AI总结 Colin通过语音和视觉模态支持儿童多层级叙述技能发展，通过生成、理解和构建阶段提升儿童的叙事能力。

GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality

GazePointAR: 一种基于情境的多模态语音助手，用于可穿戴增强现实中的代词消歧

Jaewook Lee, Jun Wang, Elizabeth Brown, Liam Chu, Sebastian S. Rodriguez, Jon E. Froehlich

专题命中 音频语音多模态 ：multimodal(title)

AI总结 GazePointAR是一种基于情境的多模态语音助手，通过结合眼动、指向手势和对话历史来提升可穿戴增强现实中的代词消歧能力。

Journal ref Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI 2024), Article 408, ACM

Dementia Detection using Multi-modal Methods on Audio Data

Saugat Kannojia, Anirudh Praveen, Danish Vasdev, Saket Nandedkar, Divyansh Mittal, Sarthak Kalankar, Shaurya Johari, Vipul Arora

机构 * IIT Kanpur（印度坎pur理工学院） ； Dept. of EE, IIT Kanpur（电子工程系，印度坎pur理工学院） 纠错

专题命中 音频语音多模态 ：multi-modal(title)

Comments 4 pages

Gaze-Enhanced Multimodal Turn-Taking Prediction in Triadic Conversations

Seongsil Heo, Calvin Murdock, Michael Proulx, Christi Miller

专题命中 音频语音多模态 ：multimodal(title)

Comments To appear in the Proceedings of Interspeech 2025