BioVERSE：面向多模态推理的生物医学模态与大语言模型的表示对齐
BioVERSE: Representation Alignment of Biomedical Modalities to LLMs for Multi-Modal Reasoning
AI总结：
针对生物医学基础模型与大语言模型嵌入空间割裂限制跨模态推理的问题，提出两阶段的BioVERSE方法，通过模态投影对齐和多模态指令微调实现多模态生物医学推理，在多任务上以更小配置超越更大基线模型。
摘要：
大语言模型（LLMs）和生物医学基础模型（BioFMs）的最新进展已在生物文本推理、分子建模和单细胞分析中取得优异成果，但它们仍处于互不连通的嵌入空间中，限制了跨模态推理能力。我们提出BIOVERSE（全称Biomedical Vector Embedding Realignment for Semantic Engagement，即面向语义交互的生物医学向量嵌入重对齐），这是一种两阶段方法，将预训练BioFMs适配为模态编码器，并通过轻量的、模态特定的投影层将其与LLMs对齐。该方法首先通过独立训练的投影层将每种模态对齐到共享的LLM空间，使它们能够自然互操作，随后用多模态数据执行标准指令微调，让各模态协同支持下游推理。通过将原始生物医学数据与LLMs中嵌入的知识统一，该方法可实现零样本标注、跨模态问答以及交互式、可解释的对话。在细胞类型注释、分子描述和蛋白质功能推理等各类任务中，紧凑配置的BIOVERSE性能超越了规模更大的LLM基线，同时能比现有BioFMs生成更丰富的生成式输出，为规范化的多模态生物医学推理奠定了基础。
英文摘要：
Recent advances in large language models (LLMs) and biomedical foundation models (BioFMs) have achieved strong results in biological text reasoning, molecular modeling, and single-cell analysis, yet they remain siloed in disjoint embedding spaces, limiting cross-modal reasoning. We present BIOVERSE (Biomedical Vector Embedding Realignment for Semantic Engagement), a two-stage approach that adapts pretrained BioFMs as modality encoders and aligns them with LLMs through lightweight, modality-specific projection layers. The approach first aligns each modality to a shared LLM space through independently trained projections, allowing them to interoperate naturally, and then applies standard instruction tuning with multi-modal data to bring them together for downstream reasoning. By unifying raw biomedical data with knowledge embedded in LLMs, the approach enables zero-shot annotation, cross-modal question answering, and interactive, explainable dialogue. Across tasks spanning cell-type annotation, molecular description, and protein function reasoning, compact BIOVERSE configurations surpass larger LLM baselines while enabling richer, generative outputs than existing BioFMs, establishing a foundation for principled multi-modal biomedical reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
