This paper presents an image-based visual servo control (IBVS) method for a first-person-view (FPV) quadrotor to conduct aggressive aerial tracking. There are three major challenges to maneuvering an underactuated vehicle using IBVS: (i) finding a visual feature representation that is robust to large rotations and is suited to be an optimization variable; (ii) keeping the target visible without sacrificing the robot's agility; and (iii) compensating for the rotational effects in the detected features. We propose a complete design framework to address these problems. First, we employ a rotation on $SO(3)$ to represent a spherical image feature on $S^{2}$ to gain singularity-free and second-order differentiable properties. To ensure target visibility, we formulate the IBVS as a nonlinear model predictive control (NMPC) problem with three constraints taken into account: the robot's physical limits, target visibility, and time-to-collision (TTC). Furthermore, we propose a novel attitude-compensation scheme to enable formulating the visibility constraint in the actual image plane instead of a virtual fix-orientation image plane. It guarantees that the visibility constraint is valid under large rotations. Extensive experimental results show that our method can track a fast-moving target stably and aggressively without the aid of a localization system.
Precise trajectory tracking is a crucial property for \acp{MAV} to operate in cluttered environment or under disturbances. In this paper we present a detailed comparison between two state-of-the-art model-based control techniques for \ac{MAV} trajectory tracking. A classical \ac{LMPC} is presented and compared against a more advanced \ac{NMPC} that considers the full system model. In a careful analysis we show the advantages and disadvantages of the two implementations in terms of speed and tracking performance. This is achieved by evaluating hovering performance, step response, and aggressive…
Hong Kong University of Science and Technology（香港科技大学）
In this paper, we propose a complete and robust motion planning system for the aggressive flight of autonomous quadrotors. The proposed method is built upon on a classical teach-and-repeat framework, which is widely adopted in infrastructure inspection, aerial transportation, and search-and-rescue. For these applications, human's intention is essential to decide the topological structure of the flight trajectory of the drone. However, poor teaching trajectories and changing environments prevent a simple teach-and-repeat system from being applied flexibly and robustly. In this paper, instead of…
PolyFly: Polytopic Optimal Planning for Collision-Free Cable-Suspended Aerial Payload Transportation
University of California Berkeley, Department of Electrical Engineering and Computer Sciences（加州大学伯克利分校电子工程与计算机科学系）；Worcester Polytechnic Institute, Robotics Engineering（沃思黑德技术学院机器人工程）
Aerobatic maneuvers in insect-scale flapping-wing aerial robots via deep-learned robust tube model predictive control
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology（电气工程与计算机科学系，麻省理工学院）；Department of Aeronautics and Astronautics, Massachusetts Institute of Technology（航空宇航系，麻省理工学院）
This paper presents an algorithm and the implementation of a motor schema to aid the visual localization subsystem of the ongoing EMILY project at Texas A and M University. The EMILY project aims to team an Unmanned Surface Vehicle (USV) with an Unmanned Aerial Vehicle (UAV) to augment the search and rescue of marine casualties during an emergency response phase. The USV is designed to serve as a flotation device once it reaches the victims. A live video feed from the UAV is provided to the casuality responders giving them a visual estimate of the USVs orientation and position to help with its…
Invisible Servoing: a Visual Servoing Approach with Return-Conditioned Latent Diffusion
Robotics and Mechatronics group, Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente（特文特大学机器人与机电组，电气工程、数学与计算机科学学院）；Department of Applied Mathematics, University of Twente（特文特大学应用数学系）；Department of Computer, Control and Management Engineering, Sapienza University of Rome（罗马萨皮恩扎大学计算机、控制与管理工程系）
Visual Servoing NMPC Applied to UAVs for Photovoltaic Array Inspection
University of Alicante（阿利坎特大学）；Instituto de Automática (INAUT)（自动化研究所）；Universidad Nacional de San Juan（圣胡安国立大学）；Centro de Investigaciones de Ciencias Humanas y de la Educación (CICHE)（人文与教育科学研究中心）；Universidad Tecnológica Indoamérica（印多大理工大学）
Full-Body Torque-Level Non-linear Model Predictive Control for Aerial Manipulation
Institut de Robòtica i Informàtica Industrial, CSIC-UPC（西班牙国家研究委员会-加泰罗尼亚理工大学工业机器人与信息技术研究所）；NASA-Jet Propulsion Laboratory, California Institute of Technology（美国国家航空航天局喷气推进实验室，加州理工学院）
Amazon（亚马逊公司）；West Virginia University（西弗吉尼亚大学）；Microsoft（微软公司）；Gather AI（Gather AI公司）；Mowito（Mowito公司）；Carnegie Mellon University（卡内基梅隆大学）
Many aerial robotic applications require the ability to land on moving platforms, such as delivery trucks and marine research boats. We present a method to autonomously land an Unmanned Aerial Vehicle on a moving vehicle. A visual servoing controller approaches the ground vehicle using velocity commands calculated directly in image space. The control laws generate velocity commands in all three dimensions, eliminating the need for a separate height controller. The method has shown the ability to approach and land on the moving deck in simulation, indoor and outdoor environments, and compared t…
Pegasus Simulator: An Isaac Sim Framework for Multiple Aerial Vehicles Simulation
Laboratory of Robotics and Engineering Systems, ISR/IST, University of Lisbon（机器人与工程系统实验室，ISR/IST，里斯本大学）；Robotics Institute, Carnegie Mellon University（机器人研究所，卡内基梅隆大学）
California Institute of Technology（加州理工学院）；ETH Zurich（苏黎世联邦理工学院）；Northeastern University（东北大学）
The feasibility of performing airborne and ground manipulation, perception, and reconnaissance using wheeled rovers, unmanned aerial vehicles, CubeSats, SmallSats and more have been evaluated before. Among all of these solutions, balloon-based systems possess merits that make them extremely attractive, e.g., a simple operation mechanism and endured operation time. However, there are many hurdles to overcome to achieve robust loitering performance in balloon-based applications. We attempt to identify design and control challenges, and propose a novel robotic platform that allows for the applica…
Manipulation and grasping with unmanned aerial vehicles (UAVs) currently require accurate positioning and are often executed at reduced speed to ensure successful grasps. This is due to the fact that typical UAVs can only accommodate rigid manipulators with few degrees of freedom, which limits their capability to compensate for disturbances caused by the vehicle positioning errors. Moreover, UAVs have to minimize external contact forces in order to maintain stability. Biological systems, on the other hand, exploit softness to overcome similar limitations, and leverage compliance to enable aggr…
Precise Aggressive Aerial Maneuvers with Sensorimotor Policies
Institute of Cyber-Systems and Control, College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院工业控制技术研究所）；Huzhou Institute of Zhejiang University（浙江大学湖州研究院）；Differential Robotics Technology Co., Ltd.（微分机器人技术有限公司）
IIIT Hyderabad（印度信息技术研究所海得拉巴校区）；University of Lincoln（林肯大学）；Mathworks（Mathworks公司）
Existing deep learning based visual servoing approaches regress the relative camera pose between a pair of images. Therefore, they require a huge amount of training data and sometimes fine-tuning for adaptation to a novel scene. Furthermore, current approaches do not consider underlying geometry of the scene and rely on direct estimation of camera pose. Thus, inaccuracies in prediction of the camera pose, especially for distant goals, lead to a degradation in the servoing performance. In this paper, we propose a two-fold solution: (i) We consider optical flow as our visual features, which are …
New York University（纽约大学）；U.S. Army Research Laboratory（美国陆军研究实验室）
Autonomous Micro Aerial Vehicles (MAVs) have the potential to be employed for surveillance and monitoring tasks. By perching and staring on one or multiple locations aerial robots can save energy while concurrently increasing their overall mission time without actively flying. In this paper, we address the estimation, planning, and control problems for autonomous perching on inclined surfaces with small quadrotors using visual and inertial sensing. We focus on planning and executing of dynamically feasible trajectories to navigate and perch to a desired target location with on board sensing an…
Autonomous UAV Pipeline Near-proximity Inspection via Disturbance-Aware Predictive Visual Servoing
School of Automation, Key Laboratory of Measurement and Control of CSE, Ministry of Education, Southeast University（自动化学院、CSE测量与控制重点实验室、教育部、东南大学）；Department of Aeronautical and Automotive Engineering, Loughborough University（航空与汽车工程系、罗汉布大学）；Research Centre for Low Altitude Economy and the Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University（低空经济研究中心和航空与航空工程系、香港理工大学）
Vision based control of Unmanned Aerial Vehicles (UAVs) has been adopted by a wide range of applications due to the availability of low-cost on-board sensors and computers. Tuning such systems to work properly requires extensive domain specific experience, which limits the growth of emerging applications. Moreover, obtaining performance limits of UAV based visual servoing is difficult due to the complexity of the models used. In this paper, we propose a novel noise tolerant approach for real-time identification and tuning of visual servoing systems, based on deep neural networks (DNN) classifi…
We present a perception constrained visual predictive control (PCVPC) algorithm for quadrotors to enable aggressive flights without using any position information. Our framework leverages nonlinear model predictive control (NMPC) to formulate a constrained image-based visual servoing (IBVS) problem. The quadrotor dynamics, image dynamics, actuation constraints, and visibility constraints are taken into account to handle quadrotor maneuvers with high agility. Two main challenges of applying IBVS to agile drones are considered: (i) high sensitivity of depths to intense orientation changes, and (…
This paper proposes a systematic solution that uses an unmanned aerial vehicle (UAV) to aggressively and safely track an agile target. The solution properly handles the challenging situations where the intent of the target and the dense environments are unknown to the UAV. Our work is divided into two parts: target motion prediction and tracking trajectory planning. The target motion prediction method utilizes target observations to reliably predict the future motion of the target considering dynamic constraints. The tracking trajectory planner follows the traditional hierarchical workflow.A t…
We consider the problem of bridging the gap between geometric tracking control theory and implementation of model predictive control (MPC) for robotic systems operating on manifolds. We propose a generic on-manifold MPC formulation based on a canonical representation of the system evolving on manifolds. Then, we present a method that solves the on-manifold MPC formulation by linearizing the system along the trajectory under tracking. There are two main advantages of the proposed scheme. The first is that the linearized system leads to an equivalent error system represented by a set of minimal …
Avian-Inspired High-Precision Tracking Control for Aerial Manipulators
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Westlake University（西湖大学）；Westlake Institute for Optoelectronics（西湖光电研究院）；Future Industrial Research Centre, Westlake University（西湖大学未来产业研究中心）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖高等研究院先进技术研究所）
AI-Enabled Image-Based Hybrid Vision/Force Control of Tendon-Driven Aerial Continuum Manipulators
Department of Mechanical, Industrial, and Mechatronics Engineering, Toronto Metropolitan University（机械、工业与机电工程系，多伦多 Metropolitan 大学）；Department of Mechanical, Industrial, and Aerospace Engineering (MIAE), Concordia University（机械、工业与航空航天工程系（MIAE），康科迪亚大学）
Intention-Aware Planner for Robust and Safe Aerial Tracking
Zhejiang University（浙江大学）；Huzhou Institute, Zhejiang University（浙江大学湖州研究院）；Huzhou Laboratory of Autonomous Intelligent Systems（湖州自主智能系统实验室）；College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）；Robotics Institute of Zhejiang University（浙江大学机器人研究院）；College of Electrical Engineering, Zhejiang University（浙江大学电气工程学院）
Micro aerial vehicles are widely being researched and employed due to their relative low operation costs and high flexibility in various applications. We study the under-actuated quadrotor perching problem, designing a trajectory planner and controller which generates feasible trajectories and drives quadrotors to desired state in state space. This paper proposes a trajectory generating and tracking method for quadrotor perching that takes the advantages of reinforcement learning controller and traditional controller. The trained low-level reinforcement learning controller would manipulate qua…
Generating 6-D Trajectories for Omnidirectional Multirotor Aerial Vehicles in Cluttered Environments
School of Mechanical Engineering and Automation, Harbin Institute of Technology Shenzhen（哈尔滨工业大学（深圳）机械工程与自动化学院）；Faculty of Aerospace Engineering, Delft University of Technology（代尔夫特理工大学航空航天工程学院）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
A Map-free Deep Learning-based Framework for Gate-to-Gate Monocular Visual Navigation aboard Miniaturized Aerial Vehicles
Dalle Molle Institute for Artificial Intelligence (IDSIA)（达勒·莫勒人工智能研究所）；USI-SUPSI（瑞士意大利语区大学与瑞士应用科学大学联盟）；Integrated Systems Laboratory (IIS)（集成系统实验室）；ETHZ（苏黎世联邦理工学院）
Object tracking is a fundamental task in computer vision with broad practical applications across various domains, including traffic monitoring, robotics, and autonomous vehicle tracking. In this project, we aim to develop a sophisticated aerial vehicle system known as Track Anything Rapter (TAR), designed to detect, segment, and track objects of interest based on user-provided multimodal queries, such as text, images, and clicks. TAR utilizes cutting-edge pre-trained models like DINO, CLIP, and SAM to estimate the relative pose of the queried object. The tracking problem is approached as a Vi…
Planar-Sector LOS Guidance for Interception of Agile Targets with Lifting-Wing Quadcopters
School of Automation Science and Electrical Engineering, Beihang University（北京航空航天大学自动化科学与电气工程学院）；Research and Development Department, China Academy of Launch Vehicle Technology（中国运载火箭技术研究院研发部）
New York University, Tandon School of Engineering（纽约大学坦登工程学院）；DEVCOM Army Research Laboratory（美国陆军作战能力发展司令部陆军研究实验室）
Autonomous Micro Aerial Vehicles are deployed for a variety tasks including surveillance and monitoring. Perching and staring allow the vehicle to monitor targets without flying, saving battery power and increasing the overall mission time without the need to frequently replace batteries. This paper addresses the Active Visual Perching (AVP) control problem to autonomously perch on inclined surfaces up to $90^\circ$. Our approach generates dynamically feasible trajectories to navigate and perch on a desired target location, while taking into account actuator and Field of View (FoV) constraints…
