使用湍流缓解 Transformer 进行大气成像
Imaging through the Atmosphere using Turbulence Mitigation Transformer
AI总结:
针对大气湍流图像恢复中泛化差、难扩展和仿真数据不足的问题,提出 TMT,结合湍流物理、时间注意力与新型仿真器,显著提升真实湍流视频恢复性能。
摘要:
恢复被大气湍流扭曲的图像是远距离成像应用中普遍存在的问题。尽管现有基于深度学习的方法在特定测试条件下已展现出良好效果,但它们存在三个局限:(1)缺乏从合成训练数据到真实湍流数据的泛化能力;(2)无法扩展,因此在将该思路扩展到大量帧时会带来内存和速度挑战;(3)缺乏快速且准确的仿真器来生成用于训练神经网络的数据。本文提出湍流缓解 Transformer(TMT),明确解决这些问题。TMT 带来三项贡献:首先,TMT 通过解耦湍流退化并引入用于消除畸变的多尺度损失,显式利用湍流物理特性,从而提升有效性。其次,TMT 提出沿时间轴的新型注意力模块,以高效提取额外特征,从而改善内存和速度。第三,TMT 引入基于 Fourier sampler、时间相关性和灵活核大小的新型仿真器,从而增强合成更优训练数据的能力。TMT 优于最先进的视频恢复模型,尤其是在从合成数据到真实湍流数据的泛化方面。代码、视频和数据集可在 https://xg416.github.io/TMT 获取。
英文摘要:
Restoring images distorted by atmospheric turbulence is a ubiquitous problem in long-range imaging applications. While existing deep-learning-based methods have demonstrated promising results in specific testing conditions, they suffer from three limitations: (1) lack of generalization capability from synthetic training data to real turbulence data; (2) failure to scale, hence causing memory and speed challenges when extending the idea to a large number of frames; (3) lack of a fast and accurate simulator to generate data for training neural networks. In this paper, we introduce the turbulence mitigation transformer (TMT) that explicitly addresses these issues. TMT brings three contributions: Firstly, TMT explicitly uses turbulence physics by decoupling the turbulence degradation and introducing a multi-scale loss for removing distortion, thus improving effectiveness. Secondly, TMT presents a new attention module along the temporal axis to extract extra features efficiently, thus improving memory and speed. Thirdly, TMT introduces a new simulator based on the Fourier sampler, temporal correlation, and flexible kernel size, thus improving our capability to synthesize better training data. TMT outperforms state-of-the-art video restoration models, especially in generalizing from synthetic to real turbulence data. Code, videos, and datasets are available at \href{https://xg416.github.io/TMT}{https://xg416.github.io/TMT}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。