用于放射学报告生成的文本反演与自监督精炼方法
Textual Inversion and Self-supervised Refinement for Radiology Report Generation
AI总结：
本文提出TISR方法，通过文本反演将图像表示为伪词以消除模态差距，并利用自监督对比学习精炼伪词，从而提升放射学报告生成的保真度，在多个基线上验证了其有效性和即插即用特性。
摘要：
现有主流方法遵循编码器-解码器范式来生成放射学报告。它们侧重于改进编码器和解码器的网络结构，这导致两个缺点：忽视模态差距和忽略报告内容约束。在本文中，我们提出了文本反演与自监督精炼（TISR）方法来解决上述两个问题。具体而言，文本反演可以通过将图像表示为伪词，将文本和图像投影到同一空间，从而消除跨模态建模差距。随后，自监督精炼通过图像与文本之间的对比损失计算来精炼这些伪词，增强生成报告对图像的保真度。值得注意的是，TISR与大多数现有方法正交，即插即用。我们在两个广泛使用的公共数据集上进行了实验，并在各种基线上取得了显著改进，这证明了TISR的有效性和泛化能力。代码将很快发布。
英文摘要：
Existing mainstream approaches follow the encoder-decoder paradigm for generating radiology reports. They focus on improving the network structure of encoders and decoders, which leads to two shortcomings: overlooking the modality gap and ignoring report content constraints. In this paper, we proposed Textual Inversion and Self-supervised Refinement (TISR) to address the above two issues. Specifically, textual inversion can project text and image into the same space by representing images as pseudo words to eliminate the cross-modeling gap. Subsequently, self-supervised refinement refines these pseudo words through contrastive loss computation between images and texts, enhancing the fidelity of generated reports to images. Notably, TISR is orthogonal to most existing methods, plug-and-play. We conduct experiments on two widely-used public datasets and achieve significant improvements on various baselines, which demonstrates the effectiveness and generalization of TISR. The code will be available soon.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
