End-to-end trained convolutional neural networks have led to a breakthrough in optical flow estimation. The most recent advances focus on improving the optical flow estimation by improving the architecture and setting a new benchmark on the publicly available MPI-Sintel dataset. Instead, in this article, we investigate how deep neural networks estimate optical flow. A better understanding of how these networks function is important for (i) assessing their generalization capabilities to unseen inputs, and (ii) suggesting changes to improve their performance. For our investigation, we focus on FlowNetS, as it is the prototype of an encoder-decoder neural network for optical flow estimation. Furthermore, we use a filter identification method that has played a major role in uncovering the motion filters present in animal brains in neuropsychological research. The method shows that the filters in the deepest layer of FlowNetS are sensitive to a variety of motion patterns. Not only do we find translation filters, as demonstrated in animal brains, but thanks to the easier measurements in artificial neural networks, we even unveil dilation, rotation, and occlusion filters. Furthermore, we find similarities in the refinement part of the network and the perceptual filling-in process which occurs in the mammal primary visual cortex.
Loria(洛林计算机科学及其应用实验室);INRIA Nancy Grand Est(法国国家信息与自动化研究所南锡大东部分部);Universidad Nacional de Rosario(罗萨里奥国立大学);CIFASIS(阿根廷智能系统研究中心);CONICET(阿根廷国家科学与技术研究理事会)
Large-FOV RGBD Imaging via Structured PSF Coding in a Bio-inspired Monocentric System
State Key Laboratory of Extreme Photonics and Instrumentation, Zhejiang University(极端光物理与仪器国家重点实验室,浙江大学);DJI Technology Co. Ltd.(大疆科技有限公司);School of Artificial Intelligence and Robotics, Hunan University(人工智能与机器人学院,湖南大学);National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University(机器人视觉感知与控制技术国家工程研究中心,湖南大学)
Frequency-Aware Physics-Inspired Degradation Model for Real-World Image Super-Resolution
John Hopcroft Center for Computer Science, Shanghai Jiao Tong University(上海交通大学约翰·霍普克罗夫特计算机科学中心);Institute of Image Communication and Network Engineering,Shanghai Jiao Tong University(上海交通大学图像通信与网络工程研究所);Department of Electrical and Computer Engineering, University of Alabama(阿拉巴马大学电气与计算机工程系);State Key Lab of Advanced Optical Communication Systems and Networks, Shanghai Jiao Tong University(上海交通大学先进光通信系统与网络国家重点实验室)