Aiming at the difficulty of extracting image features and estimating the Jacobian matrix in image based visual servo, this paper proposes an image based visual servo approach with deep learning. With the powerful learning capabilities of convolutional neural networks(CNN), autonomous learning to extract features from images and fitting the nonlinear relationships from image space to task space is achieved, which can greatly facilitate the image based visual servo procedure. Based on the above ideas a two-stream network based on convolutional neural network is designed and the corresponding control scheme is proposed to realize the four degrees of freedom visual servo of the robot manipulator. Collecting images of observed target under different pose parameters of the manipulator as training samples for CNN, the trained network can be used to estimate the nonlinear relationship from 2D image space to 3D Cartesian space. The two-stream network takes the current image and the desirable image as inputs and makes them equal to guide the manipulator to the desirable pose. The effectiveness of the approach is verified with experimental results.
Visual Servoing for Robotic On-Orbit Servicing: A Survey
Space Robotics (SpaceR) Research Group, Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg(空间机器人(SpaceR)研究组,安全、可靠性与信任跨学科中心(SnT),卢森堡大学);Redwire Space Europe, Luxembourg(Redwire Space Europe,卢森堡)
Robotic Eye-in-hand Visual Servo Axially Aligning Nasopharyngeal Swabs with the Nasal Cavity
Systems Design Engineering, University of Waterloo(滑铁卢大学系统设计工程系);Mechanical and Mechatronics Engineering, University of Waterloo(滑铁卢大学机械与机电工程系);Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院人机学与机器人研究所)
End-to-end Reinforcement Learning of Robotic Manipulation with Robust Keypoints Representation
A*STAR Institute of High Performance Computing (IHPC)(新加坡科技研究局高性能计算研究所);A*STAR Institute of Information Research (IR)(新加坡科技研究局信息研究所);National University of Singapore (NUS)(新加坡国立大学);Alibaba DAMO Academy(阿里巴巴达摩院)
Towards Real-World Aerial Vision Guidance with Categorical 6D Pose Tracker
Hunan University(湖南大学);National Engineering Research Centre for Robot Visual Perception and Control(国家机器人视觉感知与控制技术工程研究中心);Nanyang Technological University(南洋理工大学)
Khalifa University Center for Autonomous Robotic Systems (KUCARS), Khalifa University of Science and Technology(哈利法大学自主机器人系统中心(KUCARS),哈利法科技大学);Kingston University(金斯顿大学);Purdue University(普渡大学)