MELINDA: A Multimodal Dataset for Biomedical Experiment Method Classification
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of Southern California（南加利福尼亚大学）；Intuit Inc.（英图伊特公司）；Chan Zuckerberg Initiative（陈·扎克伯格倡议）
We introduce a new dataset, MELINDA, for Multimodal biomEdicaL experImeNt methoD clAssification. The dataset is collected in a fully automated distant supervision manner, where the labels are obtained from an existing curated database, and the actual contents are extracted from papers associated with each of the records in the database. We benchmark various state-of-the-art NLP and computer vision models, including unimodal models which only take either caption texts or images as inputs, and multimodal models. Extensive experiments and analysis show that multimodal models, despite outperforming unimodal ones, still need improvements especially on a less-supervised way of grounding visual concepts with languages, and better transferability to low resource domains. We release our dataset and the benchmarks to facilitate future research in multimodal learning, especially to motivate targeted improvements for applications in scientific domains.
MUDI: A Multimodal Biomedical Dataset for Understanding Pharmacodynamic Drug-Drug Interactions
VNU University of Engineering and Technology (VNU-UET)（越南工程技术大学）；Lausanne University Hospital (CHUV)（洛桑大学医院）；University of Lausanne (UNIL)（洛桑大学）；Hanoi Medical University（河内医学院）；National Geriatric Hospital, Hanoi（河内国立老年医院）
A scoping review on multimodal deep learning in biomedical images and texts
Weill Cornell Medicine（威尔康奈尔医学院）；National Center for Biotechnology Information (NCBI), National Library of Medicine (NLM), National Institutes of Health (NIH)（美国国立卫生研究院国家医学图书馆国家生物技术信息中心）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Automated knowledge discovery from trending chemical literature is essential for more efficient biomedical research. How to extract detailed knowledge about chemical reactions from the core chemistry literature is a new emerging challenge that has not been well studied. In this paper, we study the new problem of fine-grained chemical entity typing, which poses interesting new challenges especially because of the complex name mentions frequently occurring in chemistry literature and graphic representation of entities. We introduce a new benchmark data set (CHEMET) to facilitate the study of the…
The recent advancement of pre-trained Transformer models has propelled the development of effective text mining models across various biomedical tasks. However, these models are primarily learned on the textual data and often lack the domain knowledge of the entities to capture the context beyond the sentence. In this study, we introduced a novel framework that enables the model to learn multi-omnics biological information about entities (proteins) with the help of additional multi-modal cues like molecular structure. Towards this, rather developing modality-specific architectures, we devise a…
Harvard T.H. Chan School of Public Health（哈佛陈曾熙公共卫生学院）；Harvard Medical School（哈佛医学院）；University of North Carolina School of Medicine（北卡罗来纳大学医学院）
Word embeddings are a popular approach to unsupervised learning of word relationships that are widely used in natural language processing. In this article, we present a new set of embeddings for medical concepts learned using an extremely large collection of multimodal medical data. Leaning on recent theoretical insights, we demonstrate how an insurance claims database of 60 million members, a collection of 20 million clinical notes, and 1.7 million full text biomedical journal articles can be combined to embed concepts into a common space, resulting in the largest ever set of embeddings for 1…
Doctor Sun: A Bilingual Multimodal Large Language Model for Biomedical AI
Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education East China University of Science and Technology（能源化工过程智能制造重点实验室，东华大学）；Research Institute of Intelligent Control and Systems Harbin Institute of Technology（智能控制与系统研究室，哈尔滨工业大学）；Department of Emergency Medicine, Sir Run Run Shaw Hospital Zhejiang University School of Medicine（浙江大学医学院急诊医学科）；Provincial Key Laboratory of Precise Diagnosis Treatment of Abdominal Infection, Sir Run Run Shaw Hospital Zhejiang University School of Medicine（腹部感染精准诊断治疗省级重点实验室，浙江大学医学院）；School of Medicine Shaoxing University（绍兴大学医学院）
Doctor Sun是一种双语多模态大语言模型，通过整合预训练视觉编码器和医学LLM，提升生物医学多模态任务的性能，并提供SunMed-VL数据集支持研究进展。
Beyond Captions: Context-Grounded Reconstruction for Biomedical Multimodal Continued Pretraining
The Hong Kong Polytechnic University（香港理工大学）；Sun Yat-sen University（中山大学）；InfiX.ai；PolyU-Daya Bay Technology and Innovation Research Institute（PolyU-大亚湾技术与创新研究院）
From Compound Figures to Medical Multi-image Reasoning: Scaling Multimodal Large Language Models with Biomedical Literature
Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University, New Haven, CT 06510, USA（耶鲁医学院生物医学信息学与数据科学系，耶鲁大学，新 Haven，CT 06510，USA）；School of Medicine, University of Puerto Rico, San Juan, PR 00921, USA（波多黎各大学医学院，波多黎各，San Juan，PR 00921，USA）
Biomedical Visual Instruction Tuning with Clinician Preference Alignment
Stanford University（斯坦福大学）；Emory University（埃默里大学）；University of California, Berkeley（加州大学伯克利分校）；University of Washington（华盛顿大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
Tabular LLMs for Interpretable Few-Shot Alzheimer's Disease Prediction with Multimodal Biomedical Data
University of Pennsylvania（宾夕法尼亚大学）；Stony Brook University（石溪大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Medical University of South Carolina（南卡罗来纳医科大学）；Cedars-Sinai Medical Center（西格拉姆山医疗中心）
Capabilities of Claude Fable 5 on Biomedical Challenge Problems
School of Computing, University of Georgia（佐治亚大学计算学院）；Department of Chemistry, University of Illinois（伊利诺伊大学化学系）；Institute of Bioinformatics, University of Georgia（佐治亚大学生物信息学研究所）
Advancing AI Research Assistants with Expert-Involved Learning
Yale University（耶鲁大学）；Broad Institute of MIT and Harvard（麻省理工学院-哈佛大学博德研究所）；Google DeepMind（谷歌DeepMind）；Stanford University（斯坦福大学）；Genentech（基因泰克）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Cornell University（康奈尔大学）；Harvard University（哈佛大学）
MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research
Stanford University（斯坦福大学）；Tsinghua University（清华大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Princeton University（普林斯顿大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）；Chan Zuckerberg Biohub Network（陈·扎克翰生物中心网络）
