对齐大型语言模型与几何深度模型以用于蛋白质表示
Aligning Large Language Models and Geometric Deep Models for Protein Representation
AI总结：
本研究探索蛋白质领域中LLM与几何深度模型的表示对齐，全面评估多种模型并提出增加GDM嵌入维度、使用两层投影头及微调LLM等策略以提升对齐质量。
摘要：
潜在表示对齐已成为构建多模态大型语言模型（MLLM）的基础技术，其通过将不同模态的嵌入映射到共享空间（通常与大型语言模型（LLM）的嵌入空间对齐）来实现有效的跨模态理解。尽管初步聚焦于蛋白质的MLLM已经出现，但它们主要依赖启发式方法，缺乏对不同表示间最佳对齐实践的基础理解。在本研究中，我们探索了蛋白质领域中LLM与几何深度模型（GDM）之间的多模态表示对齐。我们全面评估了3个最先进的LLM（Gemma2-2B、LLaMa3.1-8B和LLaMa3.1-70B）与4个蛋白质专用GDM（GearNet、GVP、ScanNet、GAT）。我们的工作从模型和蛋白质两个视角考察了对齐因素，识别了当前对齐方法中的挑战，并提出了改进对齐过程的策略。我们的主要发现表明，结合图和3D结构信息的GDM与LLM对齐效果更好，更大的LLM展现出更优的对齐能力，且蛋白质罕见度显著影响对齐性能。我们还发现，增加GDM嵌入维度、使用两层投影头以及在蛋白质特定数据上微调LLM能大幅提升对齐质量。这些策略为蛋白质相关多模态模型的性能提供了潜在提升。我们的代码和数据可在 https://github.com/Tizzzzy/LLM-GDM-alignment 获取。
英文摘要：
Latent representation alignment has become a foundational technique for constructing multimodal large language models (MLLM) by mapping embeddings from different modalities into a shared space, often aligned with the embedding space of large language models (LLMs) to enable effective cross-modal understanding. While preliminary protein-focused MLLMs have emerged, they have predominantly relied on heuristic approaches, lacking a fundamental understanding of optimal alignment practices across representations. In this study, we explore the alignment of multimodal representations between LLMs and Geometric Deep Models (GDMs) in the protein domain. We comprehensively evaluate three state-of-the-art LLMs (Gemma2-2B, LLaMa3.1-8B, and LLaMa3.1-70B) with four protein-specialized GDMs (GearNet, GVP, ScanNet, GAT). Our work examines alignment factors from both model and protein perspectives, identifying challenges in current alignment methodologies and proposing strategies to improve the alignment process. Our key findings reveal that GDMs incorporating both graph and 3D structural information align better with LLMs, larger LLMs demonstrate improved alignment capabilities, and protein rarity significantly impacts alignment performance. We also find that increasing GDM embedding dimensions, using two-layer projection heads, and fine-tuning LLMs on protein-specific data substantially enhance alignment quality. These strategies offer potential enhancements to the performance of protein-related multimodal models. Our code and data are available at https://github.com/Tizzzzy/LLM-GDM-alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
