GTP-4o：面向全模态生物医学表征的模态提示异质图学习
GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representation
AI总结：
本文提出GTP-4o，用异质图、模态提示补全和知识引导层级聚合统一建模多种临床模态，解决模态缺失与跨模态交互问题。
摘要：
多模态表征学习的最新进展已在生物医学领域取得成功。尽管现有技术能够处理多模态信息，但由于固有的模态鸿沟，当扩展到多种临床模态以及实际的模态缺失场景时，仍面临挑战。为解决这些问题，我们提出了一种创新的用于全模态学习的模态提示异质图（GTP-4o），它将大量不同的临床模态嵌入到统一表征中，补全缺失模态的不充分嵌入，并通过基于图的聚合重新表述跨模态学习。具体而言，我们建立了异质图嵌入，以在模态特定特征（节点）和跨模态关系（边）上显式捕获多样语义属性。随后，我们设计了模态提示补全，通过图提示机制补全缺失模态的不充分图表征，该机制生成幻觉式图形拓扑，以引导缺失嵌入趋向完整表征。通过补全后的图，我们精心开发了知识引导的层级跨模态聚合，包括用于沿领域知识驱动的路径发现潜在异质邻居的全局元路径邻居模块，以及用于跨各种异质关系进行全面跨模态交互的局部多关系聚合模块。我们在严格的基准实验中对照先前的最先进方法评估了所提方法的有效性。简言之，GTP-4o初步探索了一个引人入胜的领域：通过图论整体地嵌入、关联并感知来自多种临床模态的异质模式。项目页面：https://gtp-4-o.github.io/。
英文摘要：
Recent advances in learning multi-modal representation have witnessed the success in biomedical domains. While established techniques enable handling multi-modal information, the challenges are posed when extended to various clinical modalities and practical modalitymissing setting due to the inherent modality gaps. To tackle these, we propose an innovative Modality-prompted Heterogeneous Graph for Omnimodal Learning (GTP-4o), which embeds the numerous disparate clinical modalities into a unified representation, completes the deficient embedding of missing modality and reformulates the cross-modal learning with a graph-based aggregation. Specially, we establish a heterogeneous graph embedding to explicitly capture the diverse semantic properties on both the modality-specific features (nodes) and the cross-modal relations (edges). Then, we design a modality-prompted completion that enables completing the inadequate graph representation of missing modality through a graph prompting mechanism, which generates hallucination graphic topologies to steer the missing embedding towards the intact representation. Through the completed graph, we meticulously develop a knowledge-guided hierarchical cross-modal aggregation consisting of a global meta-path neighbouring to uncover the potential heterogeneous neighbors along the pathways driven by domain knowledge, and a local multi-relation aggregation module for the comprehensive cross-modal interaction across various heterogeneous relations. We assess the efficacy of our methodology on rigorous benchmarking experiments against prior state-of-the-arts. In a nutshell, GTP-4o presents an initial foray into the intriguing realm of embedding, relating and perceiving the heterogeneous patterns from various clinical modalities holistically via a graph theory. Project page: https://gtp-4-o.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
