FCN-LLM: 通过图级多任务指令微调赋能LLM理解脑功能连接网络
FCN-LLM: Empower LLM for Brain Functional Connectivity Network Understanding via Graph-level Multi-task Instruction Tuning
AI总结:
FCN-LLM通过图级多任务指令微调使LLM理解脑功能连接网络,提升其在临床任务中的泛化能力。
摘要:
大型语言模型在语言理解和推理方面取得了显著成功,其多模态扩展能够理解图像、视频和音频。受此启发,源自静息态fMRI的脑功能连接网络基础模型在临床任务中显示出潜力。然而,现有方法未能将FCN与文本模态对齐,限制了LLM直接理解FCN的能力。为此,我们提出FCN-LLM框架,通过图级多任务指令微调使LLM理解FCN。我们的方法采用多尺度FCN编码器,捕捉脑区、功能子网络和全脑特征,并将其投影到LLM的语义空间中。我们设计涵盖19种主体特性的多范式指令任务,覆盖人口学、表型和精神疾病方面。多阶段学习策略首先对齐FCN嵌入与LLM,然后联合微调整个模型以捕捉高层语义信息。在大规模、多中心FCN数据库上的实验表明,FCN-LLM在未见数据集上实现了强大的零样本泛化能力,优于传统监督和基础模型。本工作引入了将脑功能网络与LLM整合的新范式,为神经科学提供了一个灵活且可解释的框架。
英文摘要:
Large Language Models have achieved remarkable success in language understanding and reasoning, and their multimodal extensions enable comprehension of images, video, and audio. Inspired by this, foundation models for brain functional connectivity networks derived from resting-state fMRI have shown promise in clinical tasks. However, existing methods do not align FCNs with the text modality, limiting the ability of LLMs to directly understand FCNs. To address this, we propose FCN-LLM, a framework that enables LLMs to understand FCNs through graph-level, multi-task instruction tuning. Our approach employs a multi-scale FCN encoder capturing brain-region, functional subnetwork, and whole-brain features, projecting them into the semantic space of LLM. We design multi-paradigm instruction tasks covering 19 subject-specific attributes across demographics, phenotypes, and psychiatric conditions. A multi-stage learning strategy first aligns FCN embeddings with the LLM and then jointly fine-tunes the entire model to capture high-level semantic information. Experiments on a large-scale, multi-site FCN database show that FCN-LLM achieves strong zero-shot generalization on unseen datasets, outperforming conventional supervised and foundation models. This work introduces a new paradigm for integrating brain functional networks with LLMs, offering a flexible and interpretable framework for neuroscience.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。