多模态大模型

Rank-Based Modeling for Universal Packets Compression in Multi-Modal Communications

Xuanhao Luo, Zhiyuan Peng, Zhouyu Li, Ruozhou Yu, Yuchen Liu

专题命中 音频语音多模态 ：multi-modal(title)

Comments Accepted for publication in 26th IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM)

Music and art: a study in cross-modal interpretation

Paul Warren, Paul Mulholland, Naomi Barker

专题命中 音频语音多模态 ：cross-modal(title)

Emotion-Aware Interaction Design in Intelligent User Interface Using Multi-Modal Deep Learning

Shiyu Duan, Ziyi Wang, Shixiao Wang, Mengmeng Chen, Runsheng Zhang

专题命中 音频语音多模态 ：multi-modal(title)

When Robots Get Chatty: Grounding Multimodal Human-Robot Conversation and Collaboration

Philipp Allgeuer, Hassan Ali, Stefan Wermter

专题命中 音频语音多模态 ：multimodal(title)

Journal ref International Conference on Artificial Neural Networks, Sep 2024 (pp. 306-321)

Multimodal photometry and spectroscopy: a new approach to data analysis in Astrophysics

Johanna Casado, Beatriz García

专题命中 音频语音多模态 ：multimodal(title)

Comments 22 pages, links to sonification files and videos, submitted to RASTI

Integrated Control of Robotic Arm through EMG and Speech: Decision-Driven Multimodal Data Fusion

Tauheed Khan Mohd, Ahmad Y Javaid

专题命中 音频语音多模态 ：multimodal(title)

3D audio-visual recordings of mosquito wings for aeroacoustic simulation

Lionel Feugère, Jung-Hee Seo, Umair Ismail, Gabriella Gibson, Rajat Mittal

专题命中 音频语音多模态 ：audio-visual(title)

Journal ref Forum Acusticum, Sep 2023, Torino, Italy

Antagonising explanation and revealing bias directly through sequencing and multimodal inference

Luís Arandas, Mick Grierson, Miguel Carvalhais

专题命中 音频语音多模态 ：multimodal(title)

Comments 3 pages, no figures. ACM C&C 23 Workshop paper

A neuro-symbolic approach for multimodal reference expression comprehension

Aman Jain, Anirudh Reddy Kondapally, Kentaro Yamada, Hitomi Yanaka

专题命中 音频语音多模态 ：multimodal(title)

Comments Appeared in the 37th Annual Conference of the Japanese Society for Artificial Intelligence, 2023

AVscript: Accessible Video Editing with Audio-Visual Scripts

Mina Huh, Saelyne Yang, Yi-Hao Peng, Xiang 'Anthony' Chen, Young-Ho Kim, Amy Pavel

专题命中 音频语音多模态 ：audio-visual(title)

Comments CHI 2023

A multimodal approach for Parkinson disease analysis

Marcos Faundez-Zanuy, Antonio Satue-Villar, Jiri Mekyska, Viridiana Arreola, Pilar Sanz, Carles Paul, Luis Guirao, Mateu Serra, Laia Rofes, Pere Clavé, Enric Sesa-Nogueras, Josep Roure

专题命中 音频语音多模态 ：multimodal(title)

Comments 10 pages

Journal ref In: Bassis S., Esposito A., Morabito F. (eds) Advances in Neural Networks: Computational and Theoretical Issues. Smart Innovation, Systems and Technologies, vol 37. Springer, Cham. 2015

Multi-modal fusion with gating using audio, lexical and disfluency features for Alzheimer's Dementia recognition from spontaneous speech

Morteza Rohanian, Julian Hough, Matthew Purver

专题命中 音频语音多模态 ：multi-modal(title)

Journal ref Proc. Interspeech 2020, 2187-2191

Natural interaction with traffic control cameras through multimodal interfaces

Marco Grazioso, Alessandro Sebastian Podda, Silvio Barra, Francesco Cutugno

专题命中 音频语音多模态 ：multimodal(title)

Journal ref AI-HCI Conference 2021

Multimodal Continuous Emotion Recognition using Deep Multi-Task Learning with Correlation Loss

Berkay Köprü, Engin Erzin

专题命中 音频语音多模态 ：multimodal(title)

Comments 6 pages,letter

Cross-modal variational inference for bijective signal-symbol translation

Axel Chemla--Romeu-Santos, Stavros Ntalampiras, Philippe Esling, Goffredo Haus, Gérard Assayag

专题命中 音频语音多模态 ：cross-modal(title)

Comments Proceedings of the 22nd International Conference on Digital Audio Effects (DAFx-2019)

Model enhancement and personalization using weakly supervised learning for multi-modal mobile sensing

Diyan Teng, Rashmi Kulkarni, Justin McGloin

专题命中 音频语音多模态 ：multi-modal(title)

Adaptive Grasp Control through Multi-Modal Interactions for Assistive Prosthetic Devices

Michelle Esponda, Thomas M. Howard

专题命中 音频语音多模态 ：multi-modal(title)

Comments Presented at AI-HRI AAAI-FSS, 2018 (arXiv:1809.06606)

Synthesis of Tongue Motion and Acoustics from Text using a Multimodal Articulatory Database

Ingmar Steiner, Sébastien Le Maguer, Alexander Hewer

专题命中 音频语音多模态 ：multimodal(title)

Journal ref IEEE/ACM Transactions on Audio, Speech, and Language Processing 25 (2017) 2351 - 2361

Multi-Modal Hybrid Deep Neural Network for Speech Enhancement

Zhenzhou Wu, Sunil Sivadas, Yong Kiam Tan, Ma Bin, Rick Siow Mong Goh

专题命中 音频语音多模态 ：multi-modal(title)

GEMINI: A Generic Multi-Modal Natural Interface Framework for Videogames

Luís Filipe Teófilo, Pedro Alves Nogueira, Pedro Brandão Silva

专题命中 音频语音多模态 ：multi-modal(title)

Comments WorldCIST'13 Internacional Conference

Using multimodal speech production data to evaluate articulatory animation for audiovisual speech synthesis

Ingmar Steiner, Korin Richmond, Slim Ouni

专题命中 音频语音多模态 ：multimodal(title)

Journal ref 3rd International Symposium on Facial Analysis and Animation (2012)

TurboT2VA: Fast Large-Scale Text-to-Video-Audio Generation via Score-Regularized Consistency Distillation

TurboT2VA：基于分数正则化一致性蒸馏的快速大规模文本-视频-音频生成

Xiaoda Yang, Yuxiang Liu, Kaiwen Zheng, Yuan Liu, Yibo Lai, Shengpeng Ji, Kai Jiang, Jianfei Chen, Xiaobin Hu, Shuicheng Yan, Jintao Zhang, Jun Zhu, Zhou Zhao

机构 * Zhejiang University（浙江大学） ； Tianjin University（天津大学） ； Tsinghua University（清华大学） ； Qingdao University（青岛大学） ； National University of Singapore（新加坡国立大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

AI总结 本文提出TurboT2VA框架，通过多模态归一化与渐进式课程方案，在保持音视频生成质量的同时，大幅加速190亿参数联合T2VA模型的推理，在不同分辨率下实现最高54.67倍的生成器加速。

EXAM$^2$: $\underline{Ex}tending$ $\underline{A}udio$ $Understanding$ $in$ $\underline{M}ultilingual$ $and$ $\underline{M}ultimodal$ $Analysis$

EXAM²：扩展多语言与多模态分析中的音频理解能力

Jiawen Wang, Xiaoxue Gao, Zi Haur Pang, Nancy F. Chen

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.AI

AI总结 本文提出多语言多模态音频理解基准EXAM²，评估现有模型发现其存在多语言跨模态理解差距，微调的Gemma3n-EXAM²性能显著提升，推动相关研究发展。

Comments 8 pages, 2 figures

A Survey of Advancing Audio Super-Resolution and Bandwidth Extension from Discriminative to Generative Models

音频超分辨率和带宽扩展从判别模型到生成模型的综述

Ningyuan Yang, Yize Li, Diego A. Cuji, Ryan M. Corey, Pu Zhao, Xue Lin, Andrew C. Singer

机构 * Discovery Partners Institute（发现伙伴研究所） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；multimodal foundation model(abstract)；分类 eess.AS

AI总结 本文综述了音频超分辨率和带宽扩展领域，从判别模型向生成模型的转变，总结了早期判别模型的局限性以及生成模型在表示域、架构、条件机制等方面的改进，探讨了大语言模型和多模态基础模型等新兴方向，并指出了感知评估、相位建模和实际应用泛化等开放挑战。

Comments Under review

VOICE: A Vision-Omics Foundation Model Integrating Direct and Retrieval-Based Prediction of In-situ Single-Cell Gene Expression

VOICE：一种融合原位单细胞基因表达直接预测与基于检索预测的视觉-组学基础模型

Xin Luo, Yicheng Tao, Haoxuan Zeng, Suyuan Wang, Chenzi Ouyang, Meiqi Zhu, Kai Liu, Shuibing Chen, Jie Liu

机构 * University of Michigan（密歇根大学） ； Weill Cornell Medicine（威尔康奈尔医学院） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；multimodal foundation model(abstract)；分类 cs.CV

AI总结 该研究提出多模态基础模型 VOICE，通过对比学习对齐 H&E 形态与单细胞表达嵌入，融合直接回归与参考检索分支，泛化性良好且在七项指标上优于现有方法，可从 H&E 图像预测单细胞基因表达。

AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward

AVCap：用细节感知奖励强化音视频联合字幕

Mingyang Wu, Kaituo Feng, Bohao Li, Kaixiong Gong, Zihao Yin, Xiangyu Yue

机构 * MMLab, CUHK（香港中文大学MMLab） ； Peking University（北京大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV

AI总结 针对音视频联合字幕的数据集、奖励信号、评估基准的局限，提出AVCap-100K数据集、AVCap模型及专用基准指标，通过Da-GRPO优化的AVCap模型性能优异。

AnyTrack: Unifying Visual Object Tracking with Any Modalities

AnyTrack：用任意模态统一视觉目标跟踪

Hao Li, Yunzhi Zhuge, Wenning Hao, Pingping Zhang, Xiaoxiong Zhang, Dong Wang, Huchuan Lu

机构 * Army Engineering University of PLA（中国人民解放军陆军工程大学） ； Dalian University of Technology（大连理工大学） ； National University of Defense Technology（国防科技大学） 纠错

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV

AI总结 针对现有多模态目标跟踪器适配性与泛化性差的问题，提出AnyTrack框架，通过MIM与CUM模块实现任意模态的统一跟踪，扩展基准后实验验证其性能先进、灵活有效。

Comments Accepted by ACM MM2026. More modifications may be performed

EmpaAva: An Open-source Agentic 3D-Avatar Empathetic Live Chatbot

EmpaAva：开源智能体式3D化身共情实时聊天机器人

Jie Yang, Wenhao Xu, Shuhui Lin, Hao Fei

机构 * National University of Singapore（新加坡国立大学） ； Tsinghua University（清华大学） ； University of Oxford（牛津大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CL

AI总结 该研究提出首个开源智能体式3D化身共情聊天机器人EmpaAva，通过三智能体架构实现闭环多模态交互，在评估中优于多款基准模型，将开源并提供在线演示。

Comments Project&Demo: https://empaava.top/

HyPASE: Hyperbolic Geometry for Parameter-Efficient Speech Emotion Fine-Tuning Framework for Large Audio-Language Models

HyPASE：用于大音频语言模型的参数高效语音情感微调框架的双曲几何方法

Tian Jin, Ruikang Zhang, Zefeng Zhao, Ding Luo, Jin Zeng

机构 * Tongji University（同济大学） ； The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）） ； Peking University（北京大学） 纠错

专题命中 音频语音多模态 ：cross-modal(abstract,abstract_cn)；分类 cs.AI

AI总结 本文提出HyPASE双曲PEFT框架，利用庞加莱球模型及HGA、EMCA组件，在MELD、IEMOCAP等数据集上优于欧氏PEFT基线，实现高效的大音频语言模型语音情感微调。

OPOD: On-Policy Omni Distillation

OPOD：基于策略的全模态蒸馏

Tong Zhao, Yuyang Hu, Yutao Zhu, Reed Li, Haijin Liang, Haibo Shi, Yu Lu, Zhicheng Dou

机构 * Tencent（腾讯） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；omni-modal(abstract)；分类 cs.AI

AI总结 研究旨在提升全模态模型能力，提出基于策略的全模态蒸馏（OPOD）方法，将学生响应路由到对应模态教师，调整教师影响，训练后仅留可部署的全模态模型，在多基准测试和多种骨干网络规模下取得优异成绩。