The computer vision and robotics research communities are each strong. However progress in computer vision has become turbo-charged in recent years due to big data, GPU computing, novel learning algorithms and a very effective research methodology. By comparison, progress in robotics seems slower. It is true that robotics came later to exploring the potential of learning -- the advantages over the well-established body of knowledge in dynamics, kinematics, planning and control is still being debated, although reinforcement learning seems to offer real potential. However, the rapid development of computer vision compared to robotics cannot be only attributed to the former's adoption of deep learning. In this paper, we argue that the gains in computer vision are due to research methodology -- evaluation under strict constraints versus experiments; bold numbers versus videos.
University of Washington（华盛顿大学）；Cornell University（康奈尔大学）；NVIDIA（英伟达）
Building perceptual systems for robotics which perform well under tight computational budgets requires novel architectures which rethink the traditional computer vision pipeline. Modern vision architectures require the agent to build a summary representation of the entire scene, even if most of the input is irrelevant to the agent's current goal. In this work, we flip this paradigm, by introducing EarlyFusion vision models that condition on a goal to build custom representations for downstream tasks. We show that these goal specific representations can be learned more quickly, are substantiall…
Wearable and Assistive robotics for human grasp support are broadly either tele-operated robotic arms or act through orthotic control of a paralyzed user's hand. Such devices require correct orientation for successful and efficient grasping. In many human-robot assistive settings, the end-user is required to explicitly control the many degrees of freedom making effective or efficient control problematic. Here we are demonstrating the off-loading of low-level control of assistive robotics and active orthotics, through automatic end-effector orientation control for grasping. This paper describes…
Rutgers University（罗格斯大学）；Cleveland State University（克利夫兰州立大学）
New efficient source feature compression solutions are proposed based on a two-stage Walsh-Hadamard Transform (WHT) for Convolutional Neural Network (CNN)-based object classification in underwater robotics. The object images are firstly transformed by WHT following a two-stage process. The transform-domain tensors have large values concentrated in the upper left corner of the matrices in the RGB channels. By observing this property, the transform-domain matrix is partitioned into inner and outer regions. Consequently, two novel partitioning methods are proposed in this work: (i) fixing the siz…
EarthSense Inc.（EarthSense公司）；University of Illinois at Urbana Champaign（伊利诺伊大学厄巴纳-香槟分校）
We report promising results for high-throughput on-field soybean pod count with small mobile robots and machine-vision algorithms. Our results show that the machine-vision based soybean pod counts are strongly correlated with soybean yield. While pod counts has a strong correlation with soybean yield, pod counting is extremely labor intensive, and has been difficult to automate. Our results establish that an autonomous robot equipped with vision sensors can autonomously collect soybean data at maturity. Machine-vision algorithms can be used to estimate pod-counts across a large diversity panel…
This paper describes the design of a multi-camera optical tactile sensor that provides information about the contact force distribution applied to its soft surface. This information is contained in the motion of spherical particles spread within the surface, which deforms when subject to force. The small embedded cameras capture images of the different particle patterns that are then mapped to the three-dimensional contact force distribution through a machine learning architecture. The design proposed in this paper exhibits a larger contact surface and a thinner structure than most of the exis…
TCS Research & Innovation, Tata Consultancy Services（塔塔咨询服务公司TCS研究与创新部）
Creating a vision pipeline for different datasets to solve a computer vision task is a complex and time consuming process. Currently, these pipelines are developed with the help of domain experts. Moreover, there is no systematic structure to construct a vision pipeline apart from relying on experience, trial and error or using template-based approaches. As the search space for choosing suitable algorithms for achieving a particular vision task is large, human exploration for finding a good solution requires time and effort. To address the following issues, we propose a dynamic and data-driven…
Vision Language Action Models in Robotic Manipulation: A Systematic Review
Khalifa University Center for Autonomous Robotic Systems (KUCARS), Khalifa University, United Arab Emirates（卡利法大学自主机器人系统中心（KUCARS）、卡利法大学、阿拉伯联合酋长国）；Institute of Industrial and Control Engineering (IOC), Universitat Politecnica de Catalunya, Spain（工业与控制工程研究所（IOC）、巴塞罗那技术大学、西班牙）
Due to the trending need of building autonomous robotic perception system, sensor fusion has attracted a lot of attention amongst researchers and engineers to make best use of cross-modality information. However, in order to build a robotic platform at scale we need to emphasize on autonomous robot platform bring-up cost as well. Cameras and radars, which inherently includes complementary perception information, has potential for developing autonomous robotic platform at scale. However, there is a limited work around radar fused with Vision, compared to LiDAR fused with vision work. In this pa…
Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；School of Psychological and Cognitive Sciences, Peking University（北京大学心理与认知科学学院）；Beijing Key Laboratory of Behavior and Mental Health, Peking University（北京大学行为与心理健康北京市重点实验室）；State Key Lab of General AI, Beijing Institute for General AI（人工智能通用状态关键实验室，北京人工智能研究院）；AIVC Laboratory, University of California, Los Angeles（加州大学洛杉矶分校AIVC实验室）；Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence（北京大学武汉人工智能研究院具身智能实验室）
University of Oslo（奥斯陆大学）；Graz University of Technology（格拉茨技术大学）
Electric vehicles (EVs) and plug-in hybrid vehicles (PHEVs) are rapidly gaining popularity on our roads. Besides a comparatively high purchasing price, the main two problems limiting their use are the short driving range and inconvenient charging process. In this paper we address the following by presenting an automatic robot-based charging station with 3D vision guidance for plugging and unplugging the charger. First of all, the whole system concept consisting of a 3D vision system, an UR10 robot and a charging station is presented. Then we show the shape-based matching methods used to succes…
JingDong Group（京东集团）；Tongji University（同济大学）；Technische Universität München（慕尼黑工业大学）；ETH Zurich（苏黎世联邦理工学院）
Robotic grasping plays an important role in the field of robotics. The current state-of-the-art robotic grasping detection systems are usually built on the conventional vision, such as RGB-D camera. Compared to traditional frame-based computer vision, neuromorphic vision is a small and young community of research. Currently, there are limited event-based datasets due to the troublesome annotation of the asynchronous event stream. Annotating large scale vision dataset often takes lots of computation resources, especially the troublesome data for video-level annotation. In this work, we consider…
BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulation
Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences（人工智能安全重点实验室，计算技术研究所，中国科学院，中国科学院大学）
Teleoperation offers the possibility of imparting robotic systems with sophisticated reasoning skills, intuition, and creativity to perform tasks. However, current teleoperation solutions for high degree-of-actuation (DoA), multi-fingered robots are generally cost-prohibitive, while low-cost offerings usually provide reduced degrees of control. Herein, a low-cost, vision based teleoperation system, DexPilot, was developed that allows for complete control over the full 23 DoA robotic system by merely observing the bare human hand. DexPilot enables operators to carry out a variety of complex man…
ARC Centre of Excellence for Robotic Vision (ACRV)（ARC机器人视觉卓越中心（ACRV））；Queensland University of Technology (QUT)（昆士兰科技大学（QUT））
This paper introduces a machine learning based system for controlling a robotic manipulator with visual perception only. The capability to autonomously learn robot controllers solely from raw-pixel images and without any prior knowledge of configuration is shown for the first time. We build upon the success of recent deep reinforcement learning and develop a system for learning target reaching with a three-joint robot manipulator using external visual observation. A Deep Q Network (DQN) was demonstrated to perform target reaching after training in simulation. Transferring the network to real h…
Dresden University of Applied Sciences（德累斯顿应用技术大学）
Machine vision is critical to robotics due to a wide range of applications which rely on input from visual sensors such as autonomous mobile robots and smart production systems. To create the smart homes and systems of tomorrow, an overview about current challenges in the research field would be of use to identify further possible directions, created in a systematic and reproducible manner. In this work a systematic literature review was conducted covering research from the last 10 years. We screened 172 papers from four databases and selected 52 relevant papers. While robustness and computati…
Robotic Grasping of Harvested Tomato Trusses Using Vision and Online Learning
Delft University of Technology（代尔夫特理工大学）；Czech Technical University in Prague（布拉格捷克技术大学）；Czech Institute of Informatics, Robotics, and Cybernetics（捷克信息学、机器人学与控制论研究所）
VLA-4D: Embedding 4D Awareness into Vision-Language-Action Models for SpatioTemporally Coherent Robotic Manipulation
School of Computing, National University of Singapore（新加坡国立大学计算机学院）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）
Google Brain（谷歌大脑）；University of California Berkeley（加州大学伯克利分校）
In this paper, we study the problem of learning vision-based dynamic manipulation skills using a scalable reinforcement learning approach. We study this problem in the context of grasping, a longstanding challenge in robotic manipulation. In contrast to static learning behaviors that choose a grasp point and then execute the desired grasp, our method enables closed-loop vision-based control, whereby the robot continuously updates its grasp strategy based on the most recent observations to optimize long-horizon grasp success. To that end, we introduce QT-Opt, a scalable self-supervised vision-b…
Georgia Institute of Technology（佐治亚理工学院）；Twentieth Century Fox（二十世纪福克斯公司）；Starfleet Academy（星际舰队学院）
Deep reinforcement learning (RL) algorithms can learn complex robotic skills from raw sensory inputs, but have yet to achieve the kind of broad generalization and applicability demonstrated by deep learning methods in supervised domains. We present a deep RL method that is practical for real-world robotics tasks, such as robotic manipulation, and generalizes effectively to never-before-seen tasks and objects. In these settings, ground truth reward signals are typically unavailable, and we therefore propose a self-supervised model-based approach, where a predictive model learns to directly pred…
ARC Centre of Excellence for Robotic Vision（澳大利亚研究理事会机器人视觉卓越中心）；Monash University（莫纳什大学）
State-of-the-art deep neural network recognition systems are designed for a static and closed world. It is usually assumed that the distribution at test time will be the same as the distribution during training. As a result, classifiers are forced to categorise observations into one out of a set of predefined semantic classes. Robotic problems are dynamic and open world; a robot will likely observe objects that are from outside of the training set distribution. Classifier outputs in robotic applications can lead to real-world robotic action and as such, a practical recognition system should no…
A Vision-Based Shared-Control Teleoperation Scheme for Controlling the Robotic Arm of a Four-Legged Robot
Department of Mechanical Engineering, University of São Paulo（机械工程系，圣保罗大学）；Instituto Israelita de Ensino e Pesquisa Albert Einstein, Hospital Israelita Albert Einstein（艾利哈德研究所，以色列医院）
Intel Corporation, Santa Clara, CA, USA（美国加利福尼亚州圣克拉拉市英特尔公司）
Traditional image signal processors (ISPs) are primarily designed and optimized to improve the image quality perceived by humans. However, optimal perceptual image quality does not always translate into optimal performance for computer vision applications. We propose a set of methods, which we collectively call VisionISP, to repurpose the ISP for machine consumption. VisionISP significantly reduces data transmission needs by reducing the bit-depth and resolution while preserving the relevant information. The blocks in VisionISP are simple, content-aware, and trainable. Experimental results sho…
Computer vision approaches are widely used by autonomous robotic systems to sense the world around them and to guide their decision making as they perform diverse tasks such as collision avoidance, search and rescue, and object manipulation. High accuracy is critical, particularly for Human-on-the-loop (HoTL) systems where decisions are made autonomously by the system, and humans play only a supervisory role. Failures of the vision model can lead to erroneous decisions with potentially life or death consequences. In this paper, we propose a solution based upon adaptive autonomy levels, whereby…
This paper presents a robotics vision-based heuristic reasoning system for underwater target tracking and navigation. This system is introduced to improve the level of automation of underwater Remote Operated Vehicles (ROVs) operations. A prototype which combines computer vision with an underwater robotics system is successfully designed and developed to perform target tracking and intelligent navigation. ...
University of Waterloo（滑铁卢大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
There has been increasing interest in smart factories powered by robotics systems to tackle repetitive, laborious tasks. One impactful yet challenging task in robotics-powered smart factory applications is robotic grasping: using robotic arms to grasp objects autonomously in different settings. Robotic grasping requires a variety of computer vision tasks such as object detection, segmentation, grasp prediction, pick planning, etc. While significant progress has been made in leveraging of machine learning for robotic grasping, particularly with deep learning, a big challenge remains in the need…
This paper presents a comprehensive survey on vision-based robotic grasping. We conclude three key tasks during vision-based robotic grasping, which are object localization, object pose estimation and grasp estimation. In detail, the object localization task contains object localization without classification, object detection and object instance segmentation. This task provides the regions of the target object in the input data. The object pose estimation task mainly refers to estimating the 6D object pose and includes correspondence-based methods, template-based methods and voting-based meth…
This paper describes the vision based robotic picking system that was developed by our team, Team Applied Robotics, for the Amazon Picking Challenge 2016. This competition challenged teams to develop a robotic system that is able to pick a large variety of products from a shelve or a tote. We discuss the design considerations and our strategy, the high resolution 3D vision system, the use of a combination of texture and shape-based object detection algorithms, the robot path planning and object manipulators that were developed.
Universidade Estadual do Ceará（塞阿拉州立大学）；Universidade de Fortaleza (UNIFOR)（福塔莱萨大学（UNIFOR））
This work was conceived based on Project-Based Learning (ABP) and presents the design, development and mathematical modeling steps of a low-cost robotic manipulator with five degrees of freedom through an interdisciplinary project linking two very important disciplines of the course of Control Engineering and Automation of the University of Fortaleza: Computer Aided Design, Microcontrollers and Industrial Robotics. At the end are presented the results that the project has brought to the best learning of the discipline on the optics of the tutor and students.
Texas A & M University（德州农工大学）；Wayne State University（韦恩州立大学）；Carnegie Mellon University（卡内基梅隆大学）；Simon Fraser University（西蒙弗雷泽大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Rice University（莱斯大学）；University of North Carolina-Chapel Hill（北卡罗来纳大学教堂山分校）
Proceedings of the 1st Workshop on Robotics Challenges and Vision (RCV2013)
Effective methods of preventing falls significantly improve the quality of life of the Elderly. Nowadays, people focus mainly on the proper provision of the apartment with handrails and fall detection systems once they have occurred. The article presents a~system of active detection and classification of the risk of falls in the home space using a~service robot equipped with a~vision sensor. Hazard classification allows for effective performance of tasks assigned to the robot while maintaining a~high level of user safety.
Bio-inspired reconfigurable stereo vision for robotics using omnidirectional cameras
Shenzhen Key Laboratory of Intelligent Robotics and Flexible Manufacturing Systems, Southern University of Science and Technology（深圳智能机器人与柔性制造系统重点实验室，南方科技大学）
Jacobs University Bremen（不来梅雅各布大学）；Universitat de Girona（赫罗纳大学）
Use of robotic infrastructures can significantly increase with remote access. This would open up the possibility to use costly equipment without the need to buy them, or to simply access those assets remotely when actual travel is not possible or recommended - for example in pandemic times. In this paper we present an analysis of several networking techniques which allow remote robotics operations, alongside with experimental results with distance ranging from hundreds of meters up to thousands of kilometers.
