arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.00619cs.CVcs.AIcs.LG
收藏

生物医学图像分析中上下文长度与高效Transformer的研究

A Study on Context Length and Efficient Transformers for Biomedical Image Analysis

Office of AI Research, NHLBI, NIH(美国国立卫生研究院心肺血液研究所人工智能研究办公室);Health Futures, Microsoft Research(微软研究院健康未来部门)

Sarah M. Hooper, Hui Xue

AI总结:

本文系统评估了上下文长度对生物医学图像分析的影响,发现其与性能强相关,并验证了长上下文模型在保持性能的同时能显著提升效率。

摘要:

生物医学成像模态通常产生高分辨率、多维图像,这给深度神经网络带来了计算挑战。由于自注意力算子的计算复杂度随上下文长度呈二次方增长,在训练Transformer时这些计算挑战进一步加剧。长上下文模型的最新进展有潜力缓解这些困难,并使Transformer能够更高效地应用于大型生物医学图像,但目前缺乏针对该主题的系统性评估。在本研究中,我们考察了上下文长度对生物医学图像分析的影响,并评估了最近提出的长上下文模型的性能。我们首先整理了一套生物医学成像数据集,包括用于分割、去噪和分类任务的2D和3D数据。然后,我们通过改变图像块大小和注意力窗口大小,使用Vision Transformer和Swin Transformer分析了上下文长度对网络性能的影响。我们的发现揭示了上下文长度与性能之间的强相关性,尤其是在像素级预测任务中。最后,我们表明最近的长上下文模型在保持可比性能的同时,在效率方面表现出显著提升,尽管我们也指出了仍然存在的差距。这项工作强调了在生物医学成像中使用长上下文模型的潜力和挑战。

英文摘要:

Biomedical imaging modalities often produce high-resolution, multi-dimensional images that pose computational challenges for deep neural networks. These computational challenges are compounded when training transformers due to the self-attention operator, which scales quadratically with context length. Recent developments in long-context models have potential to alleviate these difficulties and enable more efficient application of transformers to large biomedical images, although a systematic evaluation on this topic is lacking. In this study, we investigate the impact of context length on biomedical image analysis and we evaluate the performance of recently proposed long-context models. We first curate a suite of biomedical imaging datasets, including 2D and 3D data for segmentation, denoising, and classification tasks. We then analyze the impact of context length on network performance using the Vision Transformer and Swin Transformer by varying patch size and attention window size. Our findings reveal a strong relationship between context length and performance, particularly for pixel-level prediction tasks. Finally, we show that recent long-context models demonstrate significant improvements in efficiency while maintaining comparable performance, though we highlight where gaps remain. This work underscores the potential and challenges of using long-context models in biomedical imaging.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑