U-VLM：用于报告生成的分层视觉语言模型
U-VLM: Hierarchical Vision Language Modeling for Report Generation
AI总结：
U-VLM通过分层视觉语言建模在CT-RATE和AbdomenAtlas 3.0上实现了最先进的报告生成性能，展示了精心设计的视觉编码器预训练的重要性。
摘要：
自动化的放射科报告生成对于减少放射科医生的工作量和提高诊断一致性至关重要，但生成3D医学影像的准确报告仍然具有挑战性。现有的视觉-语言模型面临两个限制：它们没有利用分割预训练的编码器，且它们只在语言模型的输入层注入视觉特征，从而丢失了多尺度信息。我们提出了U-VLM，它在训练和架构上都实现了分层的视觉-语言建模：（1）从分割到分类再到报告生成的逐步训练，以及（2）多层视觉注入，将U-Net编码器特征路由到相应的语言模型层。每个训练阶段都可以利用不同的数据集，而无需统一的标注。U-VLM仅使用从头开始训练的0.1B解码器，在CT-RATE（F1：0.414 vs 0.258，BLEU-mean：0.349 vs 0.305）和AbdomenAtlas 3.0（F1：0.624 vs 0.518用于基于分割的检测）上实现了最先进的性能，证明了精心设计的视觉编码器预训练优于7B+预训练语言模型。消融研究显示，逐步预训练显著提高了F1，而多层注入提高了BLEU-mean。代码可在https://github.com/yinghemedical/U-VLM上获得。
英文摘要：
Automated radiology report generation is key for reducing radiologist workload and improving diagnostic consistency, yet generating accurate reports for 3D medical imaging remains challenging. Existing vision-language models face two limitations: they do not leverage segmentation-pretrained encoders, and they inject visual features only at the input layer of language models, losing multi-scale information. We propose U-VLM, which enables hierarchical vision-language modeling in both training and architecture: (1) progressive training from segmentation to classification to report generation, and (2) multi-layer visual injection that routes U-Net encoder features to corresponding language model layers. Each training stage can leverage different datasets without unified annotations. U-VLM achieves state-of-the-art performance on CT-RATE (F1: 0.414 vs 0.258, BLEU-mean: 0.349 vs 0.305) and AbdomenAtlas 3.0 (F1: 0.624 vs 0.518 for segmentation-based detection) using only a 0.1B decoder trained from scratch, demonstrating that well-designed vision encoder pretraining outweighs the benefits of 7B+ pre-trained language models. Ablation studies show that progressive pretraining significantly improves F1, while multi-layer injection improves BLEU-mean. Code is available at https://github.com/yinghemedical/U-VLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
