多模态大模型

SocialPulse: On-Device Detection of Social Interactions in Naturalistic Settings Using Smartwatch Sensing

SocialPulse: 使用智能手表多模态感知在自然环境中检测社交互动

Md Sabbir Ahmed, Kaitlyn Dorothy Petz, Noah French, Tanvi Lakhtakia, Aayushi Sangani, Mark Rucker, Xinyu Chen, Bethany A. Teachman, Laura E. Barnes

专题命中 音频语音多模态 ：multimodal(title)

AI总结 本文提出了一种在设备上检测自然环境中社交互动的方法，通过智能手表的多模态感知技术，实现了对多种社交互动的准确识别，并在真实场景中验证了其有效性。

Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings

赋予传感器声音：用于语义时间序列嵌入的多模态JEPA

Utsav Dutta, Gerardo Pastrana, Sina Khoshfetrat Pakazad, Henrik Ohlsson

机构 * C3 AI 纠错

专题命中 音频语音多模态 ：multimodal(title)

AI总结 提出CHARM模型，通过通道级文本描述与Transformer编码器结合，利用联合嵌入预测架构（JEPA）学习语义时间序列嵌入，在异常检测、分类和预测任务中仅用线性探针即取得强性能。

Comments 9 pages, 5 figures, accepted at ICML 2026. arXiv admin note: substantial text overlap with arXiv:2505.14543

Journal ref Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 2026

GuardReasoner-Omni: A Reasoning-based Multi-modal Guardrail for Text, Image, Video, and Audio

GuardReasoner-Omni：一种基于推理的多模态护栏，适用于文本、图像、视频和音频

Zhenhao Zhu, Yue Liu, Yanpei Guo, Wenjie Qu, Cancan Chen, Yufei He, Yibo Li, Yulin Chen, Tianyi Wu, Huiying Xu, Xinzhong Zhu, Jiaheng Zhang

专题命中 音频语音多模态 ：multi-modal(title)

AI总结 提出GuardReasoner-Omni，一种基于推理的多模态护栏模型，通过两阶段训练（SFT和RL）在3B和7B参数规模上实现对文本、图像、视频和音频数据的有效审核，性能优于现有基线。

Gesture First, LLM-Assisted Voice Complement: Exploring Multimodal Robot 'Puppeteer' Teleoperation Via Virtual Counterpart in Augmented Reality

先手势，LLM辅助语音补充：通过增强现实中的虚拟对应物探索多模态机器人'提线人'遥控

Yuchong Zhang, Bastian Orthmann, Shichen Ji, Michael Welle, Jonne Van Haastregt, Danica Kragic

机构 * KTH Royal Institute of Technology（皇家理工学院） 纠错

专题命中 音频语音多模态 ：multimodal(title)

AI总结 本文探讨了通过增强现实中的虚拟对应物实现多模态机器人遥控的方法，比较了仅手势和结合语音与手势的交互方式在性能和用户体验上的差异，提出设计指南以平衡效率、鲁棒性和用户专业性。

Comments This work is under peer review

Feelings, Not Feel: Affective Audio-Visual Pseudo-Haptics in Hand-Tracked XR

感受，而非感受：手部追踪XR中的情感音频视觉伪触觉

Kristian Paolo David, Tyrone Justin Sta Maria, Mikkel Dominic Gamboa, Jordan Aiko Deja

专题命中 音频语音多模态 ：audio-visual(title)

AI总结 该研究探讨了在无控制器的XR环境中，通过伪触觉线索（音频、视觉）影响用户情感体验的机制，发现其更应视为情感反馈而非直接触觉替代。

Comments 5 pages, 2 figures, 1 table, CHI EA Posters

Journal ref Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems (CHI EA '26), April 13--17, 2026, Barcelona, Spain

Dynamite: Real-Time Debriefing Slide Authoring through AI-Enhanced Multimodal Interaction

Dynamite：通过AI增强的多模态交互实现实时复盘幻灯片创作

Panayu Keelawat, David Barron, Kaushik Narasimhan, Daniel Manesh, Xiaohang Tang, Xi Chen, Sang Won Lee, Yan Chen

专题命中 音频语音多模态 ：multimodal(title)

AI总结 Dynamite通过AI增强的多模态交互实现实时复盘幻灯片创作，提升课堂讨论后复盘的准确性和效率。

Comments Accepted to VL/HCC 2025

Generative Multi-modal Feedback for Singing Voice Synthesis Evaluation

生成式多模态反馈用于歌唱语音合成评估

Xueyan Li, Yuxin Wang, Mengjie Jiang, Qingzi Zhu, Jiang Zhang, Zoey Kim, Yazhe Niu

机构 * Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） ； University of Science and Technology of China（中国科学技术大学） ； Columbia University（哥伦比亚大学） ； Dalian University of Technology（大连理工大学） ； The Chinese University of Hong Kong（香港中文大学） 纠错

专题命中 音频语音多模态 ：multi-modal(title)

AI总结 本文提出生成式多模态反馈框架，通过音频-语言模型生成多维语言和音频反馈，提升歌唱语音合成评估的准确性和可解释性。

Comments 16 pages, 5 figures

Colin: A Multimodal Human-AI Co-Creation Storytelling System To Support Children's Multi-Level Narrative Skills

Colin：一种多模态人机协同创故事叙述系统，以支持儿童的多层级叙述技能

Lyumanshan Ye, Jiandong Jiang, Yuhan Liu, Yihan Ran, Yufan Zhou, Zhao Wang, Yipeng Yu, Pengfei Liu, Danni Chang, Yucheng Jin

专题命中 音频语音多模态 ：multimodal(title)

AI总结 Colin通过语音和视觉模态支持儿童多层级叙述技能发展，通过生成、理解和构建阶段提升儿童的叙事能力。

GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality

GazePointAR: 一种基于情境的多模态语音助手，用于可穿戴增强现实中的代词消歧

Jaewook Lee, Jun Wang, Elizabeth Brown, Liam Chu, Sebastian S. Rodriguez, Jon E. Froehlich

专题命中 音频语音多模态 ：multimodal(title)

AI总结 GazePointAR是一种基于情境的多模态语音助手，通过结合眼动、指向手势和对话历史来提升可穿戴增强现实中的代词消歧能力。

Journal ref Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI 2024), Article 408, ACM

Dementia Detection using Multi-modal Methods on Audio Data

Saugat Kannojia, Anirudh Praveen, Danish Vasdev, Saket Nandedkar, Divyansh Mittal, Sarthak Kalankar, Shaurya Johari, Vipul Arora

机构 * IIT Kanpur（印度坎pur理工学院） ； Dept. of EE, IIT Kanpur（电子工程系，印度坎pur理工学院） 纠错

专题命中 音频语音多模态 ：multi-modal(title)

Comments 4 pages

Gaze-Enhanced Multimodal Turn-Taking Prediction in Triadic Conversations

Seongsil Heo, Calvin Murdock, Michael Proulx, Christi Miller

专题命中 音频语音多模态 ：multimodal(title)

Comments To appear in the Proceedings of Interspeech 2025

Rank-Based Modeling for Universal Packets Compression in Multi-Modal Communications

Xuanhao Luo, Zhiyuan Peng, Zhouyu Li, Ruozhou Yu, Yuchen Liu

专题命中 音频语音多模态 ：multi-modal(title)

Comments Accepted for publication in 26th IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM)

Music and art: a study in cross-modal interpretation

Paul Warren, Paul Mulholland, Naomi Barker

专题命中 音频语音多模态 ：cross-modal(title)

Emotion-Aware Interaction Design in Intelligent User Interface Using Multi-Modal Deep Learning

Shiyu Duan, Ziyi Wang, Shixiao Wang, Mengmeng Chen, Runsheng Zhang

专题命中 音频语音多模态 ：multi-modal(title)

When Robots Get Chatty: Grounding Multimodal Human-Robot Conversation and Collaboration

Philipp Allgeuer, Hassan Ali, Stefan Wermter

专题命中 音频语音多模态 ：multimodal(title)

Journal ref International Conference on Artificial Neural Networks, Sep 2024 (pp. 306-321)

Multimodal photometry and spectroscopy: a new approach to data analysis in Astrophysics

Johanna Casado, Beatriz García

专题命中 音频语音多模态 ：multimodal(title)

Comments 22 pages, links to sonification files and videos, submitted to RASTI

Integrated Control of Robotic Arm through EMG and Speech: Decision-Driven Multimodal Data Fusion

Tauheed Khan Mohd, Ahmad Y Javaid

专题命中 音频语音多模态 ：multimodal(title)

3D audio-visual recordings of mosquito wings for aeroacoustic simulation

Lionel Feugère, Jung-Hee Seo, Umair Ismail, Gabriella Gibson, Rajat Mittal

专题命中 音频语音多模态 ：audio-visual(title)

Journal ref Forum Acusticum, Sep 2023, Torino, Italy

Antagonising explanation and revealing bias directly through sequencing and multimodal inference

Luís Arandas, Mick Grierson, Miguel Carvalhais

专题命中 音频语音多模态 ：multimodal(title)

Comments 3 pages, no figures. ACM C&C 23 Workshop paper

A neuro-symbolic approach for multimodal reference expression comprehension

Aman Jain, Anirudh Reddy Kondapally, Kentaro Yamada, Hitomi Yanaka

专题命中 音频语音多模态 ：multimodal(title)

Comments Appeared in the 37th Annual Conference of the Japanese Society for Artificial Intelligence, 2023

AVscript: Accessible Video Editing with Audio-Visual Scripts

Mina Huh, Saelyne Yang, Yi-Hao Peng, Xiang 'Anthony' Chen, Young-Ho Kim, Amy Pavel

专题命中 音频语音多模态 ：audio-visual(title)

Comments CHI 2023

A multimodal approach for Parkinson disease analysis

Marcos Faundez-Zanuy, Antonio Satue-Villar, Jiri Mekyska, Viridiana Arreola, Pilar Sanz, Carles Paul, Luis Guirao, Mateu Serra, Laia Rofes, Pere Clavé, Enric Sesa-Nogueras, Josep Roure

专题命中 音频语音多模态 ：multimodal(title)

Comments 10 pages

Journal ref In: Bassis S., Esposito A., Morabito F. (eds) Advances in Neural Networks: Computational and Theoretical Issues. Smart Innovation, Systems and Technologies, vol 37. Springer, Cham. 2015

Multi-modal fusion with gating using audio, lexical and disfluency features for Alzheimer's Dementia recognition from spontaneous speech

Morteza Rohanian, Julian Hough, Matthew Purver

专题命中 音频语音多模态 ：multi-modal(title)

Journal ref Proc. Interspeech 2020, 2187-2191

Natural interaction with traffic control cameras through multimodal interfaces

Marco Grazioso, Alessandro Sebastian Podda, Silvio Barra, Francesco Cutugno

专题命中 音频语音多模态 ：multimodal(title)

Journal ref AI-HCI Conference 2021

Multimodal Continuous Emotion Recognition using Deep Multi-Task Learning with Correlation Loss

Berkay Köprü, Engin Erzin

专题命中 音频语音多模态 ：multimodal(title)

Comments 6 pages,letter

Cross-modal variational inference for bijective signal-symbol translation

Axel Chemla--Romeu-Santos, Stavros Ntalampiras, Philippe Esling, Goffredo Haus, Gérard Assayag

专题命中 音频语音多模态 ：cross-modal(title)

Comments Proceedings of the 22nd International Conference on Digital Audio Effects (DAFx-2019)

Model enhancement and personalization using weakly supervised learning for multi-modal mobile sensing

Diyan Teng, Rashmi Kulkarni, Justin McGloin

专题命中 音频语音多模态 ：multi-modal(title)

Adaptive Grasp Control through Multi-Modal Interactions for Assistive Prosthetic Devices

Michelle Esponda, Thomas M. Howard

专题命中 音频语音多模态 ：multi-modal(title)

Comments Presented at AI-HRI AAAI-FSS, 2018 (arXiv:1809.06606)

Synthesis of Tongue Motion and Acoustics from Text using a Multimodal Articulatory Database

Ingmar Steiner, Sébastien Le Maguer, Alexander Hewer

专题命中 音频语音多模态 ：multimodal(title)

Journal ref IEEE/ACM Transactions on Audio, Speech, and Language Processing 25 (2017) 2351 - 2361

Multi-Modal Hybrid Deep Neural Network for Speech Enhancement

Zhenzhou Wu, Sunil Sivadas, Yong Kiam Tan, Ma Bin, Rick Siow Mong Goh

专题命中 音频语音多模态 ：multi-modal(title)