生成以定位：多模态文本条件化提升医学视觉-语言模型中的短语定位
Generate to Ground: Multimodal Text Conditioning Boosts Phrase Grounding in Medical Vision-Language Models
AI总结：
提出利用生成式扩散模型结合冻结的领域特定语言模型进行零样本医学短语定位，并引入BBM后处理技术对齐文本与图像偏差，使mIoU翻倍，显著提升了疾病定位精度。
摘要：
短语定位，即将自然语言短语映射到特定图像区域，通过临床报告在医学图像的疾病定位方面具有巨大潜力。尽管当前最先进的方法依赖于判别式的自监督对比模型，但我们证明了利用交叉注意力图的生成式文本到图像扩散模型，能够实现更优的零样本短语定位性能。与先前的假设相反，我们展示了使用冻结的领域特定语言模型（如CXR-BERT）对扩散模型进行微调，其表现显著优于领域无关的同类模型。该设置取得了显著改进，其mIoU分数翻倍，达到了当前判别式方法的两倍。这些发现突显了生成式模型在短语定位任务中尚未充分探索的潜力。为进一步提升性能，我们引入了Bimodal Bias Merging (BBM)，这是一种新颖的后处理技术，通过对齐文本和图像偏差来识别高确定性区域。BBM优化了交叉注意力图，实现了更高的定位精度。我们的结果确立了生成式方法作为医学图像领域短语定位的更有效范式，为临床实践中更鲁棒和可解释的应用铺平了道路。源代码和模型权重可在https://github.com/Felix-012/generate_to_ground获取。
英文摘要：
Phrase grounding, i.e., mapping natural language phrases to specific image regions, holds significant potential for disease localization in medical imaging through clinical reports. While current state-of-the-art methods rely on discriminative, self-supervised contrastive models, we demonstrate that generative text-to-image diffusion models, leveraging cross-attention maps, can achieve superior zero-shot phrase grounding performance. Contrary to prior assumptions, we show that fine-tuning diffusion models with a frozen, domain-specific language model, such as CXR-BERT, substantially outperforms domain-agnostic counterparts. This setup achieves remarkable improvements, with mIoU scores doubling those of current discriminative methods. These findings highlight the underexplored potential of generative models for phrase grounding tasks. To further enhance performance, we introduce Bimodal Bias Merging (BBM), a novel post-processing technique that aligns text and image biases to identify regions of high certainty. BBM refines cross-attention maps, achieving even greater localization accuracy. Our results establish generative approaches as a more effective paradigm for phrase grounding in the medical imaging domain, paving the way for more robust and interpretable applications in clinical practice. The source code and model weights are available at https://github.com/Felix-012/generate_to_ground.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
