多模态大模型 - arXivDaily 专题

1711.08789 2018-06-14 cs.CV cs.SD eess.AS 62%

Visual Speech Enhancement

Aviv Gabbay, Asaph Shamir, Shmuel Peleg

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted to Interspeech 2018. Supplementary video: https://www.youtube.com/watch?v=nyYarDGpcYA

1805.02934 2018-05-09 cs.CV cs.SD eess.AS eess.IV 62%

Phoneme-to-viseme mappings: the good, the bad, and the ugly

Helen L Bear, Richard Harvey

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、eess.AS

Journal ref Speech Communication, Special Issue on AV expressive speech. 2017

1804.01452 2018-04-05 cs.CV cs.CL cs.SD 62%

Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input

David Harwath, Adrià Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, James Glass

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.CL

1712.00334 2017-12-04 cs.HC cs.CL cs.IR cs.LG cs.MM 62%

Enabling Embodied Analogies in Intelligent Music Systems

Fabio Paolizzo

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CL、cs.MM

Comments 4 pages

1709.05865 2017-09-19 cs.CV cs.LG cs.MM 62%

Depression Scale Recognition from Audio, Visual and Text Analysis

Shubham Dham, Anirudh Sharma, Abhinav Dhall

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.MM

1611.02879 2016-11-10 cs.CV cs.CL cs.LG 62%

Audio Visual Speech Recognition using Deep Recurrent Neural Networks

Abhinav Thanda, Shankar M Venkatesan

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.CL

1606.02562 2016-06-13 cs.AI cs.CL 62%

DialPort: Connecting the Spoken Dialog Research Community to Real User Data

Tiancheng Zhao, Kyusong Lee, Maxine Eskenazi

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、cs.AI

Comments Under Peer Review of SigDial 2016

1605.06778 2016-05-24 cs.CV cs.CL cs.IR 62%

openXBOW - Introducing the Passau Open-Source Crossmodal Bag-of-Words Toolkit

Maximilian Schmitt, Björn W. Schuller

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.CL

Comments 9 pages, 1 figure, pre-print

1603.08321 2016-03-29 cs.CV cs.CL cs.LG 62%

Audio Visual Emotion Recognition with Temporal Alignment and Perception Attention

Linlin Chao, Jianhua Tao, Minghao Yang, Ya Li, Zhengqi Wen

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.CL

1507.01209 2015-07-07 cs.CV cs.MM 62%

TV News Commercials Detection using Success based Locally Weighted Kernel Combination

Raghvendra Kannao, Prithwijit Guha

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM

1407.1165 2014-07-07 cs.CV cs.CL 62%

Recognition of Isolated Words using Zernike and MFCC features for Audio Visual Speech Recognition

Prashant Bordea, Amarsinh Varpeb, Ramesh Manzac, Pravin Yannawara

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.CL

2608.19527 2026-08-21 cs.HC cs.CL cs.SD 新提交 61%

Does Listening Matter? Backchanneling and Nodding in AI Clone

倾听重要吗？AI克隆中的反馈式回应与点头动作

Koji Inoue, Kazushi Kato, Tatsuya Kawahara, Shunichi Kasahara

机构 * Kyoto University（京都大学）； Sony Computer Science Laboratories（索尼计算机科学实验室）

专题命中音频语音多模态：multimodal(abstract,comments)；分类 cs.CL

AI总结本研究将口头反馈式回应与头部点头整合到配备语音克隆及LLM响应的AI克隆中，经35人被试内研究证实，添加互动倾听行为可提升AI克隆的感知专注度、真实感与共同在场感，其保真度需涵盖倾听行为。

Comments This paper has been accepted to the Late-Breaking Results (LBR) track of the 28th International Conference on Multimodal Interaction (ICMI 2026)

2608.18080 2026-08-20 cs.AI 新提交 61%

Large Language Models in Mental Health: A Systematic Review of Applications, Innovations, and Ethical Challenges

心理健康领域的大语言模型：应用、创新与伦理挑战的系统综述

Yisong Chen, Yifan Gao, Sijing Yu, Chuqing Zhao, Yang Lu

专题命中音频语音多模态：multimodal(abstract,comments)；分类 cs.AI

AI总结该系统综述梳理了大语言模型在心理健康领域的多类应用、技术创新，同时探讨了其面临的伦理与监管挑战，并倡导建立保障其安全公平部署的框架。

Comments Systematic review. Published in Journal of Industrial Integration and Management (2025). Applications of large language models in mental health, including social media analysis, clinical conversational agents, therapy support tools, multimodal learning, and ethical considerations

Journal ref Journal of Industrial Integration and Management (JIIM), 2025

2510.11837 2025-10-15 cs.CR cs.AI 61%

Countermind: A Multi-Layered Security Architecture for Large Language Models

Dominik Schwarz

机构 * Independent Researcher（独立研究者）

专题命中音频语音多模态：multimodal(abstract,comments)；分类 cs.AI

Comments 33 pages, 3 figures, 6 tables. Keywords: LLM security; defense-in-depth; prompt injection; activation steering; multimodal sandbox; threat modeling

2509.04736 2025-09-08 cs.CV 61%

WatchHAR: Real-time On-device Human Activity Recognition System for Smartwatches

Taeyoung Yeon, Vasco Xu, Henry Hoffmann, Karan Ahuja

机构 * Northwestern University（西北大学）； University of Chicago（芝加哥大学）

专题命中音频语音多模态：multimodal(abstract,comments)；分类 cs.CV

Comments 8 pages, 4 figures, ICMI '25 (27th International Conference on Multimodal Interaction), October 13-17, 2025, Canberra, ACT, Australia

2504.10642 2025-04-16 cs.CV 61%

SilVar-Med: A Speech-Driven Visual Language Model for Explainable Abnormality Detection in Medical Imaging

Tan-Hanh Pham, Chris Ngo, Trong-Duong Bui, Minh Luu Quang, Tan-Huong Pham, Truong-Son Hy

机构 * Florida Institute of Technology（佛罗里达理工学院）； Knovel Engineering Lab（诺维尔工程实验室）； Vietnam Military Medical University（越南军事医科大学）； Military Central Hospital（108陆军中央医院）； Can Tho University of Medicine and Pharmacy（芹苴医药大学）； University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）

专题命中音频语音多模态：multimodal(abstract,comments)；分类 cs.CV

Comments CVPR Multimodal Algorithmic Reasoning Workshop 2025 - SilVarMed

2012.13912 2020-12-29 cs.CV 61%

Exploring Emotion Features and Fusion Strategies for Audio-Video Emotion Recognition

Hengshun Zhou, Debin Meng, Yuanyuan Zhang, Xiaojiang Peng, Jun Du, Kai Wang, Yu Qiao

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CV；multimodal(comments)

Comments Accepted by ACM ICMI'19 (2019 International Conference on Multimodal Interaction)

1607.01076 2016-07-06 cs.HC cs.CV 61%

Aggressive actions and anger detection from multiple modalities using Kinect

Amol Patwardhan, Gerald Knapp

专题命中音频语音多模态：multimodal(abstract,comments)；分类 cs.CV

Comments 11 pages, 2 figures, 5 tables, in peer review with ACM TIST, Key words: Aggression, multimodal anger recognition, Kinect

cs/0211005 2009-11-30 cs.CV cs.HC 61%

Prosody Based Co-analysis for Continuous Recognition of Coverbal Gestures

Sanshzar Kettebekov, Mohammed Yeasin, Rajeev Sharma

专题命中音频语音多模态：multimodal(abstract,journal_ref)；分类 cs.CV

Comments Alternative see: http://vision.cse.psu.edu/kettebek/academ/publications.htm

Journal ref S. Kettebekov, M. Yeasin, and R. Sharma, "Prosody Based Co-analysis for Continuous Recognition of Coverbal Gestures," presented at International Conference on Multimodal Interfaces (ICMI'02), Pittsburgh, USA, 2002

2608.26431 2026-08-28 cs.SD eess.AS 新提交 57%

AudioSpan: Spanning the Duration and Depth of Audio Comprehension

AudioSpan：覆盖音频理解的时长与深度

Wen Huang, Yunfei Chu, Meng Gao, Haolin He, Jin Xu

机构 * Qwen Team, Alibaba Group（通义千问团队，阿里巴巴集团）； Tsinghua University（清华大学）； The Chinese University of Hong Kong（香港中文大学）

专题命中音频语音多模态：omni-modal(abstract)；分类 eess.AS

AI总结本研究提出覆盖10分钟至2小时音频的基准AudioSpan，含3240个分三认知层级的问题，评估12个大型音频-语言模型，发现从长音频提取相关事实是核心难点，该基准可公开获取。

2608.26153 2026-08-28 cs.AI 新提交 57%

EEG-to-Report: An Annotation and Feature-Text Framework for Training Language Models on Clinical EEG

EEG-to-Report：用于在临床脑电图上训练语言模型的标注与特征-文本框架

Xuan-The Tran, Le Trung Kien Nguyen

机构 * Vietnam Maritime University（越南海事大学）； HAISmartlink Lab（HAISmartlink实验室）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI

AI总结针对临床EEG报告依赖人工、现有工具无法满足AI训练需求的问题，提出EEG-to-Report框架，整合多格式EEG处理与标注，生成对齐特征-文本对，还含自动报告模块，为自动化EEG报告系统提供基础。

2608.24909 2026-08-27 cs.HC cs.CV cs.SD 新提交 57%

Super Star: Towards Streaming Real-time Interactive Agents for Digital Humans

超级明星：面向数字人的流式实时交互智能体

Wentao Jiang, Youchen Xie, Haidi Fan, Yajing Chen, Xin Wang, Ye Shi, Jingya Wang

机构 * ShanghaiTech University（上海科技大学）； LIGHTSPEED（腾讯光速工作室）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV

AI总结本文针对交互式数字人提出流式实时同步言语手势生成任务，构建耦合流式语音响应与在线手势生成的框架，经实验验证其在延迟-质量权衡等方面优于现有基线。

Comments Accepted by ACM Multimedia 2026. Project Page: \url{https://super-star-2026.github.io/}

2608.24579 2026-08-26 eess.AS 新提交 57%

Visually-Guided Spatial Audio Generation for $360^\circ$ In-the-Wild Speech Scenes

面向野外360°语音场景的视觉引导空间音频生成

Qingyu Luo, Peng Zhang, Wenwu Wang, Philip J. B. Jackson

专题命中音频语音多模态：audio-visual(abstract)；分类 eess.AS

AI总结针对野外360°语音场景空间音频采集质量受限问题，本文提出视觉引导的FOA语音空间化方法，构建YT-SPEECH数据集，采用Localizer-Renderer框架实现定向FOA信号重建，提升了音频相关性能。

Comments Accepted at INTERSPEECH 2026

2608.24707 2026-08-26 cs.CL 新提交 57%

Lost in Speech: Trilingual Spoken Hallucination Detection Across Audio and Transcripts

迷失在语音中：跨音频与文本的三语语音幻觉检测

Meruyert Aristombayeva, Jason S. Lucas, Chaewan Chun, Dongwon Lee

机构 * Satbayev University（萨塔巴耶夫大学）； University of Colorado Boulder（科罗拉多大学博尔德分校）； The Pennsylvania State University（宾夕法尼亚州立大学）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL

AI总结该研究针对低资源语言语音幻觉检测的空白，构建三语语音幻觉基准，评估多模态检测模型，发现文本检测更优，合成训练检测器在真实数据上迁移性强，还揭示合成基准的混淆因素。

2608.22907 2026-08-25 eess.AS 新提交 57%

Unsupervised Speech Recognition at the Syllable Level

音节级别的无监督语音识别

Liming Wang, Kai-Wei Chang, Kunio Kashino, David Harwath, Mark Hasegawa-Johnson, James R. Glass

专题命中音频语音多模态：multimodal(abstract)；分类 eess.AS

AI总结该研究针对现有无监督语音识别方法依赖G2P且训练不稳定的问题，提出基于掩码语言建模的音节级UASR框架，在LibriSpeech上使CER相对降低40%，并有效推广到低资源语言。

2608.22796 2026-08-25 eess.AS cs.SD 新提交 57%

DiaScriber: A Speech LLM for Joint Diarization and Transcription in Multi-Speaker Scenarios

DiaScriber：面向多说话人场景的联合说话人 diarization（语音分割）与转录的语音大语言模型

Bingshen Mu, Xian Shi, Xiong Wang, Zhifang Guo, Ting He, Xize Cheng, Yu Xi, Jin Xu, Lei Xie

专题命中音频语音多模态：multimodal(abstract)；分类 eess.AS

AI总结本研究提出基于Qwen3.5-Omni的端到端多说话人 diarization与转录模型DiaScriber，通过构建多样数据流程与三阶段训练策略，在多说话人场景测试集上性能优于对比方法且泛化能力出色。

2608.23189 2026-08-25 cs.CV 新提交 57%

EchoWM: Open and Enterable Omnimodal World Models

EchoWM：开放且可进入的全模态世界模型

Songchun Zhang, Yaowei Li, Junhao Zhuang, Weiyang Jin, Haoyu Wang, Xin Lu, Yilang Sun, Shiyi Zhang, Haoran Li, Xiaoxiao Ma, Yuming Li, Yijun Liu, Yaofeng Su, Yanwen Ma, Haoyu Wu, Zihan Su, Yue Ma, Lvmin Zhang, Haoyang Huang, Zeyue Xue, Anyi Rao, Nan Duan

机构 * HKUST（香港科技大学）； PKU（北京大学）； Joy Future Academy, JD（京东探索研究院）； HKU（香港大学）； THU（清华大学）； USTC（中国科学技术大学）； FDU（复旦大学）； Beihang University（北京航空航天大学）； Stanford University（斯坦福大学）

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV

AI总结该研究提出EchoWM全模态世界模型，围绕相机意图组织交互，构建互补数据引擎并采用渐进式训练等方法，在公开基准上实现强轨迹跟随与高视觉质量，支持跨视角交互及长序列多模态同步生成。

Comments 42 pages, 24 figures

2608.20433 2026-08-24 cs.RO cs.HC cs.MM cs.SD 新提交 57%

Humanoid Musical Robots as Experimental Interfaces for Music-Evoked Emotion

类人音乐机器人作为音乐诱发情绪研究的实验接口

Vincent K. M. Cheung, Jia-Yeu Lin

机构 * Sony Computer Science Laboratories（索尼计算机科学实验室）； Waseda University（早稻田大学）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.MM

AI总结本文提出将类人音乐机器人作为音乐诱发情绪研究的实验接口，通过WAS-5案例验证其技术可行性，为该领域可控研究提供方法论平台。

Comments Opinion paper accepted for presentation at the Sound and Music Computing (SMC) Conference 2026 (5-7 November in Zagreb, Croatia)

2608.17852 2026-08-19 cs.SD cs.MM 新提交 57%

UniVerse: Benchmarking and Enhancing LALMs on Culturally Inclusive Low-Resource Music Understanding

UniVerse：针对文化包容性低资源音乐理解的基准测试与增强

Ziya Zhou, Shangda Wu, Shenyang Xu, Yutong Zheng, Dafang Liang, Suin Chung, Danbinaerin Han, Junyan Jiang, Yongyi Zang, Ruibin Yuan, Rongxiu Zhong, Shilei Zhang, Junlan Feng, Jinglei Liu, Haotian Zhou, Zijin Li, Dasaem Jeong, Wei Xue, Yike Guo

机构 * Sogang University（西江大学）； KAIST（韩国科学技术院）； NYU Shanghai（上海纽约大学）； JIUTIAN Research, China Mobile（中国移动九天研究院）； The State Key Laboratory of Multimedia Information Processing, Peking University（北京大学多媒体信息处理国家重点实验室）； China Mobile (Hong Kong) Innovation Research Institute（中国移动（香港）创新研究院）； Central Conservatory of Music（中央音乐学院）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.MM

AI总结该研究针对低资源音乐理解问题，推出UniVerse基准与数据集，训练LALMs并研究多模态不平衡学习策略，实现性能提升但仍存在深层音乐理解的差距。

Comments 21 pages, 7 figures, 8 tables

2608.17812 2026-08-19 cs.MM 新提交 57%

On computational approaches to Pop music culture

流行音乐文化的计算研究方法

Arthur Flexer

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.MM

AI总结该文提出流行音乐文化的计算研究需采用多模态方法，综述相关量化研究并指出其存在的多模态方法匮乏等问题，勾勒出绘制歌词主题图谱等三个未来研究方向。

Comments 18 pages, 1 figure

1. 音频语音多模态 4597 篇

Visual Speech Enhancement

Phoneme-to-viseme mappings: the good, the bad, and the ugly

Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input

Enabling Embodied Analogies in Intelligent Music Systems

Depression Scale Recognition from Audio, Visual and Text Analysis

Audio Visual Speech Recognition using Deep Recurrent Neural Networks

DialPort: Connecting the Spoken Dialog Research Community to Real User Data

openXBOW - Introducing the Passau Open-Source Crossmodal Bag-of-Words Toolkit

Audio Visual Emotion Recognition with Temporal Alignment and Perception Attention

TV News Commercials Detection using Success based Locally Weighted Kernel Combination

Recognition of Isolated Words using Zernike and MFCC features for Audio Visual Speech Recognition

Does Listening Matter? Backchanneling and Nodding in AI Clone

Large Language Models in Mental Health: A Systematic Review of Applications, Innovations, and Ethical Challenges

Countermind: A Multi-Layered Security Architecture for Large Language Models

WatchHAR: Real-time On-device Human Activity Recognition System for Smartwatches

SilVar-Med: A Speech-Driven Visual Language Model for Explainable Abnormality Detection in Medical Imaging

Exploring Emotion Features and Fusion Strategies for Audio-Video Emotion Recognition

Aggressive actions and anger detection from multiple modalities using Kinect

Prosody Based Co-analysis for Continuous Recognition of Coverbal Gestures

AudioSpan: Spanning the Duration and Depth of Audio Comprehension

EEG-to-Report: An Annotation and Feature-Text Framework for Training Language Models on Clinical EEG

Super Star: Towards Streaming Real-time Interactive Agents for Digital Humans

Visually-Guided Spatial Audio Generation for $360^\circ$ In-the-Wild Speech Scenes

Lost in Speech: Trilingual Spoken Hallucination Detection Across Audio and Transcripts

Unsupervised Speech Recognition at the Syllable Level

DiaScriber: A Speech LLM for Joint Diarization and Transcription in Multi-Speaker Scenarios

EchoWM: Open and Enterable Omnimodal World Models

Humanoid Musical Robots as Experimental Interfaces for Music-Evoked Emotion

UniVerse: Benchmarking and Enhancing LALMs on Culturally Inclusive Low-Resource Music Understanding

On computational approaches to Pop music culture