多模态大模型

CLOP: Video-and-Language Pre-Training with Knowledge Regularizations

Guohao Li, Hu Yang, Feng He, Zhifan Feng, Yajuan Lyu, Hua Wu, Haifeng Wang

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments ACM Multimedia 2022 (MM'22)

A Dataset for Medical Instructional Video Classification and Question Answering

Deepak Gupta, Kush Attal, Dina Demner-Fushman

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding

Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, Luke Zettlemoyer

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments 9 pages, ACL Findings 2021

VLG-Net: Video-Language Graph Matching Network for Video Grounding

Mattia Soldan, Mengmeng Xu, Sisi Qu, Jesper Tegner, Bernard Ghanem

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments 14 pages, 7 figures, In proceeding of the ICCV21 workshop: AI for Creative Video Editing and Understanding 2021

Collaborative Spatial-Temporal Modeling for Language-Queried Video Actor Segmentation

Tianrui Hui, Shaofei Huang, Si Liu, Zihan Ding, Guanbin Li, Wenguan Wang, Jizhong Han, Fei Wang

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted by CVPR 2021

Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language

Hassan Akbari, Hamid Palangi, Jianwei Yang, Sudha Rao, Asli Celikyilmaz, Roland Fernandez, Paul Smolensky, Jianfeng Gao, Shih-Fu Chang

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval

Jie Lei, Licheng Yu, Tamara L. Berg, Mohit Bansal

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments ECCV 2020 (extended version, with TVC dataset+models; 35 pages)

Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA

Hyounghun Kim, Zineng Tang, Mohit Bansal

专题命中 视频多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments ACL 2020 (11 pages)

Mapping Multimodal Pilot Stress and Fatigue During Flight Sessions

飞行训练期间多模态飞行员压力与疲劳的映射

Atandrila Chowdhury, Sudip Vhaduri, Julius Keller, Debra Henneberry, Mark Wilson

专题命中 视频多模态 ：multimodal(title)

AI总结 本研究结合生理信号与主观报告，分析学员飞行员飞行训练中的压力与疲劳模式，证实数据驱动方法可用于监测飞行员健康状态。

Comments Under Review

Multimodal Spatiotemporal Atmospheric Data Assimilation with Latent Video Flow-matching

基于潜在流匹配的多模态时空大气数据同化

Dibyajyoti Chakraborty, Romit Maulik

机构 * The Pennsylvania State University（宾夕法尼亚州立大学） ； Purdue University（普渡大学） 纠错

专题命中 视频多模态 ：multimodal(title)

AI总结 该研究提出基于潜在流匹配的多模态时空大气数据同化方法，利用ERA5再分析数据训练先验，结合后验采样同化真实观测，可完成多种DA任务，从稀疏观测生成的集合预报性能达先进水平。

Mitigating Detector Ageing Effects with Graph-Based Multi-Modal Track Reconstruction at Belle II

在 Belle II 中使用基于图的多模态轨道重建减轻探测器老化效应

Lea Reuter, Tristan Brandes, Giacomo De Pietro, Torben Ferber

专题命中 视频多模态 ：multi-modal(title)

AI总结 研究 Belle II 探测器老化对轨道寻找的影响，采用基于图神经网络的算法，将轨道寻找设为全局关系聚类问题，经全探测器模拟评估性能，结果显示该算法在探测器老化时能提高鲁棒性，稳定跟踪性能。

Comments proceedings for Connecting The Dots 2025

Multimodal Empirical Bayes Variational Autoencoders for Joint Longitudinal and Time-to-Event Modeling

用于联合纵向和事件发生时间建模的多模态经验贝叶斯变分自编码器

Anders Sjöberg, Nils Olsson, Marcus Baaz, Mats Jirstrand

机构 * Fraunhofer-Chalmers Centre（弗劳恩霍夫-查尔默斯中心） ； Department of Electrical Engineering（电气工程系） ； Chalmers University of Technology（查尔姆斯理工大学） 纠错

专题命中 视频多模态 ：multimodal(title)

AI总结 研究针对纵向肿瘤测量等多源数据整合难的问题，扩展EB-VAE框架用于联合纵向和事件发生时间建模，比较不同解码器公式，纳入基因组协变量，实现肿瘤生长等联合预测，确定相关基因指标，提供了灵活概率框架。

Multimodal Smart Glove for Sign Language Recognition Using Deep Learning

基于深度学习的用于手语识别的多模态智能手套

Anh Thu Nguyen Ngoc, Tam Phong Truong, Thai Anh Nguyen Duong, Vu Linh Nguyen, Manh Duong Phung

专题命中 视频多模态 ：multimodal(title)

AI总结 研究旨在解决手语识别系统实际部署难题，提出集成可穿戴传感与深度学习的智能手套系统，利用柔性传感器、IMU及摄像头采集数据，经LSTM网络处理，准确率约95%，模型转换后可实时推理，证明系统用于手语翻译的可行性。

Comments In Proceedings of IFToMM International Symposium on Robotics and Mechatronics

M-EPDet: Real-Time Real-Bogus Classification and Transient Candidate Judgement for the EP-WXT Pipeline via Multi-Modal Data

M-EPDet：基于多模态数据的EP-WXT流水线实时真实-虚假分类与瞬变候选判断

Lang Chen, Yunfei Xu, Zhen Zhang, Dongyue Li, Hui Sun, Yuan Liu, Chenzhou Cui, Jinhui Xie, XiaoXiong Zuo, Shirui Wei, Wujun Shao

专题命中 视频多模态 ：multi-modal(title)

AI总结 提出M-EPDet三阶段框架，结合ResNet臂滤波器、双分支时-谱宇宙射线滤波器和背景感知贝叶斯块模块，实现EP-WXT数据实时候选筛选，真实源召回率98.31%，候选量减少99.25%。

Comments 20 pages, 13 figures, accepted by ApJS

TibetCPR: A Multimodal Tactile Feedback System to Enhance Cardiopulmonary Resuscitation Training in High-Altitude Regions of Tibet

TibetCPR：一种增强高海拔地区心肺复苏训练的多模态触觉反馈系统

Yibo Meng, Ruiqi Chen, Zhiming Liu, Xiaolan Ding

专题命中 视频多模态 ：multimodal(title)

AI总结 针对西藏地区心肺复苏训练资源匮乏的问题，提出低成本自引导训练系统TibetCPR，结合深度电触觉与节奏视觉反馈，实验表明能显著稳定按压节奏与深度，并提炼出自引导训练的设计原则。

Comments Accept to MobileHCI 2026

Cross-modal characterization of infant cry: validation of a chest-surface accelerometer in extracting acoustic vocal function measures

婴儿哭声的跨模态表征：胸表加速度计在提取声学发声功能测量中的验证

Winko W. An, Saketh Sundar, Lisa Yankowitz, Daryush D. Mehta, Carol L. Wilkinson

机构 * Division of Developmental Medicine, Boston Children’s Hospital（发育医学部，波士顿儿童医院） ； Harvard Medical School（哈佛医学院） ； Harvard University（哈佛大学） ； Children’s Hospital of Philadelphia（费城儿童医院） ； Center for Laryngeal Surgery and Voice Rehabilitation, Massachusetts General Hospital（嗓音康复中心，麻省总医院） 纠错

专题命中 视频多模态 ：cross-modal(title)

AI总结 本研究验证了胸表加速度计在婴儿哭声分析中的有效性，发现其能可靠捕获基频和抖动等声学特征，为噪声鲁棒且保护隐私的临床研究提供替代方案。

STQuant: Spatio-Temporal Adaptive Framework for Optimizer Quantization in Large Multimodal Model Training

STQuant：一种用于大规模多模态模型训练中优化器量化的空间-时间自适应框架

Minglu Liu, Cunchen Hu, Liangliang Xu, Fengming Tang, Ruijia Wang, Fu Yu

机构 * Xidian University（西安电子科技大学） ； China Telecom Cloud Computing Research Institute（中国电信云计算研究院） 纠错

专题命中 视频多模态 ：multimodal(title)

AI总结 STQuant通过动态分配精度降低优化器状态内存，保持模型质量，实验显示内存减少84.4%，平均比特宽低至5.1。

The First Mass Protest on Threads: Multimodal Mobilization and AI-Generated Visuals in Taiwan's Bluebird Movement

台湾蓝鸟运动中的首次大规模抗议：多模态动员与AI生成视觉内容

Tracy Weener, Ho-Chun Herbert Chang

专题命中 视频多模态 ：multimodal(title)

AI总结 本文分析台湾2024年蓝鸟运动中文本与视觉信息的传播动态，揭示算法曝光与用户支持的不对称性及AI生成图像在抗议中的作用。

Harnessing Rich Multi-Modal Data for Spatial-Temporal Homophily-Embedded Graph Learning Across Domains and Localities

利用丰富的多模态数据进行跨领域和地域的时空同质性嵌入图学习

Takuya Kurihana, Xiaojian Zhang, Wing Yee Au, Hon Yung Wong

机构 * Fujitsu Research of America（富士通美国研究机构） ； Department of Civil and Coastal Engineering University of Florida（佛罗里达大学土木与海岸工程系） 纠错

专题命中 视频多模态 ：multi-modal(title)

AI总结 本文提出一种跨领域和地域的时空同质性嵌入图学习框架，通过整合多模态数据提升城市预测性能，解决智能城市分析中的关键挑战。

Comments 18 pages, 8 figures, Presented in part at the 2025 INFORMS Annual Meeting

Sleep Apnea Detection on a Wireless Multimodal Wearable Device Without Oxygen Flow Using a Mamba-based Deep Learning Approach

基于Mamba深度学习方法的无线多模态可穿戴设备无氧流睡眠呼吸暂停检测

Dominik Luszczynski, Richard Fei Yin, Nicholas Afonin, Andrew S. P. Lim

机构 * University of Toronto Department of Medicine（多伦多大学医学系） ； Sunnybrook Research Institute Department of Medicine Neurology Div.（圣玛丽医院研究学院医学系神经病学部） 纠错

专题命中 视频多模态 ：multimodal(title)

AI总结 本文提出基于Mamba架构的深度学习模型，利用无线多模态可穿戴设备无氧流信号，实现睡眠呼吸暂停的准确检测与事件级表征。

Comments 29 pages, 14 figures. Authors Dominik Luszczynski, Richard Fei Yin and Nicholas Afonin contributed equally

Spatio-Temporal Graph Convolutional Networks for EV Charging Demand Forecasting Using Real-World Multi-Modal Data Integration

Jose Tupayachi, Mustafa C. Camur, Kevin Heaslip, Xueping Li

机构 * University of Tennessee（田纳西大学） 纠错

专题命中 视频多模态 ：multi-modal(title)

Continental-scale habitat distribution modelling with multimodal earth observation foundation models

Sara Si-Moussi, Stephan Hennekens, Sander Mucher, Stan Los, Yoann Cartier, Borja Jiménez-Alfaro, Fabio Attorre, Jens-Christian Svenning, Wilfried Thuiller

专题命中 视频多模态 ：multimodal(title)

DeepTx: Real-Time Transaction Risk Analysis via Multi-Modal Features and LLM Reasoning

Yixuan Liu, Xinlei Li, Yi Li

专题命中 视频多模态 ：multi-modal(title)

Comments Accepted to ASE'25

Research on Short-Video Platform User Decision-Making via Multimodal Temporal Modeling and Reinforcement Learning

Jinmeiyang Wang, Jing Dong, Li Zhou

专题命中 视频多模态 ：multimodal(title)

Comments 26 pages

Task-Aware Tuning of Time Constants in Spiking Neural Networks for Multimodal Classification

Chiu-Chang Cheng, Kapil Bhardwaj, Ya-Ning Chang, Sayani Majumdar, Chao-Hung Wang

专题命中 视频多模态 ：multimodal(title)

Comments 25 Pages and 5 Figures and a supplementary discussion as well

Make Silence Speak for Itself: a multi-modal learning analytic approach with neurophysiological data

Mingxuan Gao, Jingjing Chen, Yun Long, Xiaomeng Xu, Yu Zhang

专题命中 视频多模态 ：multi-modal(title)

Comments 25 pages, 6 figures

Personalized Predictions of Glioblastoma Infiltration: Mathematical Models, Physics-Informed Neural Networks and Multimodal Scans

Ray Zirui Zhang, Ivan Ezhov, Michal Balcerak, Andy Zhu, Benedikt Wiestler, Bjoern Menze, John S. Lowengrub

专题命中 视频多模态 ：multimodal(title)

Force-Constrained Visual Policy: Safe Robot-Assisted Dressing via Multi-Modal Sensing

Zhanyi Sun, Yufei Wang, David Held, Zackory Erickson

专题命中 视频多模态 ：multi-modal(title)

Cross-temporal Detection of Novel Ransomware Campaigns: A Multi-Modal Alert Approach

Sathvik Murli, Dhruv Nandakumar, Prabhat Kumar Kushwaha, Cheng Wang, Christopher Redino, Abdul Rahman, Shalini Israni, Tarun Singh, Edward Bowen

专题命中 视频多模态 ：multi-modal(title)

Comments Preprint. Under Review

CAESURA: Language Models as Multi-Modal Query Planners

Matthias Urban, Carsten Binnig

专题命中 视频多模态 ：multi-modal(title)

Comments 6 pages, 4 figures