Sketch2CT：多模态扩散用于结构感知的3D医学体积生成
Sketch2CT: Multimodal Diffusion for Structure-Aware 3D Medical Volume Generation
AI总结：
Sketch2CT通过结合用户提供的2D草图和文本描述，利用多模态扩散框架生成结构一致的3D医学体积，提升生成质量与可控性。
摘要：
扩散概率模型在生成高质量、逼真医学图像方面表现出显著潜力，为医学领域持续的数据稀缺问题提供了有希望的解决方案。然而，在多模态条件下生成具有解剖学一致结构的3D医学体积仍是一个复杂且未解决的问题。我们介绍了Sketch2CT，一种用于结构感知3D医学体积生成的多模态扩散框架，由用户提供的2D草图和捕捉3D几何语义的文本描述共同引导。该框架首先从随机噪声中生成目标器官的3D分割掩码，条件于两种模态。为有效对齐和融合这些输入，我们提出了两个关键模块，通过局部文本提示精炼草图特征，并整合全局草图-文本表示。基于胶囊-注意力主干网络，这些模块利用草图和文本的互补优势，生成解剖学准确的器官形状。合成的分割掩码随后引导潜在扩散模型进行3D CT体积合成，实现与用户定义草图和描述一致的逼真器官重建。在公共CT数据集上的广泛实验表明，Sketch2CT在生成多模态医学体积方面表现优异。其可控、低成本的生成流程使医疗数据集的原理化、高效增强成为可能。代码可在https://github.com/adlsn/Sketch2CT获取。
英文摘要：
Diffusion probabilistic models have demonstrated significant potential in generating high-quality, realistic medical images, providing a promising solution to the persistent challenge of data scarcity in the medical field. Nevertheless, producing 3D medical volumes with anatomically consistent structures under multimodal conditions remains a complex and unresolved problem. We introduce Sketch2CT, a multimodal diffusion framework for structure-aware 3D medical volume generation, jointly guided by a user-provided 2D sketch and a textual description that captures 3D geometric semantics. The framework initially generates 3D segmentation masks of the target organ from random noise, conditioned on both modalities. To effectively align and fuse these inputs, we propose two key modules that refine sketch features with localized textual cues and integrate global sketch-text representations. Built upon a capsule-attention backbone, these modules leverage the complementary strengths of sketches and text to produce anatomically accurate organ shapes. The synthesized segmentation masks subsequently guide a latent diffusion model for 3D CT volume synthesis, enabling realistic reconstruction of organ appearances that are consistent with user-defined sketches and descriptions. Extensive experiments on public CT datasets demonstrate that Sketch2CT achieves superior performance in generating multimodal medical volumes. Its controllable, low-cost generation pipeline enables principled, efficient augmentation of medical datasets. Code is available at https://github.com/adlsn/Sketch2CT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
