End-to-end optimization, which simultaneously optimizes optics and algorithms, has emerged as a powerful data-driven method for computational imaging system design. This method achieves joint optimization through backpropagation by incorporating differentiable optics simulators to generate measurements and algorithms to extract information from measurements. However, due to high computational costs, it is challenging to model both aberration and diffraction in light transport for end-to-end optimization of compound optics. Therefore, most existing methods compromise physical accuracy by neglecting wave optics effects or off-axis aberrations, which raises concerns about the robustness of the resulting designs. In this paper, we propose a differentiable optics simulator that efficiently models both aberration and diffraction for compound optics. Using the simulator, we conduct end-to-end optimization on scene reconstruction and classification. Experimental results demonstrate that both lenses and algorithms adopt different configurations depending on whether wave optics is modeled. We also show that systems optimized without wave optics suffer from performance degradation when wave optics effects are introduced during testing. These findings underscore the importance of accurate wave optics modeling in optimizing imaging systems for robust, high-performance applications.
FourierNets enable the design of highly non-local optical encoders for computational imaging
HHMI Janelia Research Campus(霍华德·休斯医学研究所珍利亚研究园区);Wuhan National Laboratory for Optoelectronics, HUST(华中科技大学武汉国家光电研究中心);Institut de la Vision, Sorbonne Université, CNRS(索邦大学法国国家科学研究中心视觉研究所);Harvard University(哈佛大学);Stanford University(斯坦福大学)
Shadow Reduction in Ultrasound Imaging Using Differentiable Simulation and Radiance Field Decomposition
University of Oxford(牛津大学);University of Amsterdam(阿姆斯特丹大学);Friedrich-Alexander-Universität Erlangen-Nürnberg(弗里德里希-亚历山大-埃尔朗根-纽伦堡大学);Imperial College London(伦敦帝国理工学院);University of Copenhagen(哥本哈根大学)
Fast Auto-Differentiable Digitally Reconstructed Radiographs for Solving Inverse Problems in Intraoperative Imaging
Harvard-MIT Health Sciences and Technology(哈佛-麻省理工卫生科学与技术部);Massachusetts Institute of Technology(麻省理工学院);Computer Science and Artificial Intelligence Laboratory(计算机科学与人工智能实验室)
NSFlow: End-to-End Differentiable Neuro-Symbolic Optical Flow for Visual Odometry
State Key Laboratory of Intelligent Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology(华中科技大学机械科学与工程学院智能制造装备与技术国家重点实验室)