ACE-LoRA：图注意上下文增强用于医疗视觉-语言模型的参数高效适应
ACE-LoRA: Graph-Attentive Context Enhancement for Parameter-Efficient Adaptation of Medical Vision-Language Models
AI总结：
本文提出ACE-LoRA框架，通过整合LoRA模块和ACE-HGNN模块，提升医疗VLM的零样本泛化能力，解决领域特异性与泛化能力的平衡问题。
摘要：
CLIP-like视觉-语言模型在自然图像上的成功促使了医疗领域的应用，但现有方法大多处于两个极端：单领域训练的专家模型在领域特定细节上表现良好但泛化能力差，而多领域训练的通用医疗VLM在语义上广泛但细粒度诊断线索被稀释。为解决这一专精与泛化之间的权衡问题，本文提出ACE-LoRA，一种适用于通用医疗VLM的参数高效适应框架，旨在保持强大的零样本泛化能力。ACE-LoRA将LoRA模块整合到冻结的图像-文本编码器中，并引入基于注意力的上下文增强超图神经网络（ACE-HGNN）模块，该模块能够捕捉超越成对相似性的高阶上下文交互，以丰富全局表示并增强局部诊断线索，从而解决先前参数高效微调（PEFT）方法忽视细粒度细节的关键限制。为进一步增强跨模态对齐，本文提出了一种标签引导的InfoNCE损失，以有效抑制语义相关图像-文本对之间的假阴性。尽管仅添加了0.95M可训练参数，ACE-LoRA在零样本分类、分割和检测基准测试中，跨多个领域均优于最先进的医疗VLM和PEFT基线。我们的代码可在https://github.com/icon-lab/ACE-LoRA上获取。
英文摘要：
The success of CLIP-like vision-language models (VLMs) on natural images has inspired medical counterparts, yet existing approaches largely fall into two extremes: specialist models trained on single-domain data, which capture domain-specific details but generalize poorly, and generalist medical VLMs trained on multi-domain data, which retain broad semantics but dilute fine-grained diagnostic cues. Bridging this specialization-generalization trade-off remains challenging. To address this problem, we propose ACE-LoRA, a parameter-efficient adaptation framework for generalist medical VLMs that maintains robust zero-shot generalization. ACE-LoRA integrates Low-Rank Adaptation (LoRA) modules into frozen image-text encoders and introduces an Attention-based Context Enhancement Hypergraph Neural Network (ACE-HGNN) module that captures higher-order contextual interactions beyond pairwise similarity to enrich global representations with localized diagnostic cues, addressing a key limitation of prior Parameter-Efficient Fine-Tuning (PEFT) methods that overlook fine-grained details. To further enhance cross-modal alignment, we formulate a label-guided InfoNCE loss to effectively suppress false negatives between semantically related image-text pairs. Despite adding only 0.95M trainable parameters, ACE-LoRA consistently outperforms state-of-the-art medical VLMs and PEFT baselines across zero-shot classification, segmentation, and detection benchmarks spanning multiple domains. Our code is available at https://github.com/icon-lab/ACE-LoRA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
