Multimodal Scientific Learning Beyond Diffusions and Flows
Graduate Group in Applied Mathematics and Computational Science(应用数学与计算科学联合研究生组);Department of Computer and Information Science(计算机与信息科学系);Department of Mechanical Engineering and Applied Mechanics(机械工程与应用力学系)
Leonardo Ferreira Guilhoto, Akshat Kaushal, Paris Perdikaris
Scientific machine learning (SciML) increasingly requires models that capture multimodal conditional uncertainty arising from ill-posed inverse problems, multistability, and chaotic dynamics. While recent work has favored highly expressive implicit generative models such as diffusion and flow-based methods, these approaches are often data-hungry, computationally costly, and misaligned with the structured solution spaces frequently found in scientific problems. We demonstrate that Mixture Density Networks (MDNs) provide a principled yet largely overlooked alternative for multimodal uncertainty quantification in SciML. As explicit parametric density estimators, MDNs impose an inductive bias tailored to low-dimensional, multimodal physics, enabling direct global allocation of probability mass across distinct solution branches. This structure delivers strong data efficiency, allowing reliable recovery of separated modes in regimes where scientific data is scarce. We formalize these insights through a unified probabilistic framework contrasting explicit and implicit distribution networks, and demonstrate empirically that MDNs achieve superior generalization, interpretability, and sample efficiency across a range of inverse, multistable, and chaotic scientific regression tasks.
Multimodal Deep Learning for Scientific Imaging Interpretation
Science and Technology Division, Corning Incorporated(康宁公司科技事业部);Robert Frederick Smith School of Chemical and Biomolecular Engineering, Cornell University(康奈尔大学罗伯特·弗雷德里克·史密斯化学与生物分子工程学院)
本文提出多模态深度学习框架 GlassLLaVA,结合论文图文数据与 GPT-4,实现玻璃材料 SEM 图像的类人解释、特征识别和缺陷检测,并建立可对照研究答案的评估指标。
MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems
Department of Computer Science, Yale University(耶鲁大学计算机科学系);Department of Computer Science, Brown University(布朗大学计算机科学系);School of Data Science, Fudan University(复旦大学数据科学学院);Shanghai Key Laboratory of Data Science(上海数据科学关键实验室);Datawiz LLC
MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research
Stanford University(斯坦福大学);Tsinghua University(清华大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Princeton University(普林斯顿大学);KTH Royal Institute of Technology(瑞典皇家理工学院);Chan Zuckerberg Biohub Network(陈·扎克翰生物中心网络)
Center for Computing Research, Sandia National Laboratories(桑迪亚国家实验室计算研究中心);Applied Information Sciences Center, Sandia National Laboratories(桑迪亚国家实验室应用信息科学中心);School of Computing and Augmented Intelligence, Arizona State University(亚利桑那州立大学计算与增强智能学院);School of Engineering and Applied Science, University of Pennsylvania(宾夕法尼亚大学工程与应用科学学院)
Probing the limitations of multimodal language models for chemistry and materials research
Friedrich Schiller University Jena(弗里德里希·席勒大学耶拿分校);Indian Institute of Technology Delhi(印度理工学院德里分校);Center for Energy and Environmental Chemistry Jena (CEEC Jena)(耶拿能源与环境化学中心);Helmholtz Institute for Polymers in Energy Applications Jena (HIPOLE Jena)(耶拿亥姆霍兹能源应用聚合物研究所);Jena Center for Soft Matter (JCSM)(耶拿软物质中心)
Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning
Berlin Institute of Health, Charité - Universitätsmedizin Berlin(柏林健康研究院,柏林夏里特大学医学中心);Department of Mathematics and Computer Science, Freie Universität Berlin(柏林自由大学数学与计算机科学系);Intelligent Medicine Institute, Fudan University(复旦大学智能医学研究院)
Department of Electronics, Information and Bioengineering, Politecnico di Milano(电子工程与信息学院,米兰理工学院);Osservatorio Astronomico di Roma, INAF(罗马天文台,INAF);AstroAI, Center for Astrophysics | | Harvard & Smithsonian(AstroAI,哈佛与史密松尼天体物理中心);Center for Computational Astrophysics, Institute for Advanced Study/The Flatiron Institute(计算天文学中心,高级研究院/Flatiron研究所)
SciTS: Scientific Time Series Understanding and Generation with LLMs
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tsinghua University(清华大学);Harbin Institute of Technology(哈尔滨工业大学);University of Science and Technology of China(中国科学技术大学)
Scientifically-Interpretable Reasoning Network (ScIReN): Discovering Hidden Relationships in the Carbon Cycle and Beyond
Department of Informatics and Intelligent Systems, Institute of Energy and the Environment, The Pennsylvania State University(信息学与智能系统系,能源与环境研究所,宾夕法尼亚州立大学)
Earth Science Foundation Models: From Perception to Reasoning and Discovery
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University(数据科学与人工智能系,香港理工大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)