This paper introduces CLIPSwarm, a new algorithm designed to automate the modeling of swarm drone formations based on natural language. The algorithm begins by enriching a provided word, to compose a text prompt that serves as input to an iterative approach to find the formation that best matches the provided word. The algorithm iteratively refines formations of robots to align with the textual description, employing different steps for "exploration" and "exploitation". Our framework is currently evaluated on simple formation targets, limited to contour shapes. A formation is visually represented through alpha-shape contours and the most representative color is automatically found for the input word. To measure the similarity between the description and the visual representation of the formation, we use CLIP [1], encoding text and images into vectors and assessing their similarity. Subsequently, the algorithm rearranges the formation to visually represent the word more effectively, within the given constraints of available drones. Control actions are then assigned to the drones, ensuring robotic behavior and collision-free movement. Experimental results demonstrate the system's efficacy in accurately modeling robot formations from natural language descriptions. The algorithm's versatility is showcased through the execution of drone shows in photorealistic simulation with varying shapes. We refer the reader to the supplementary video for a visual reference of the results.
From Drone Imagery to Livability Mapping: AI-powered Environment Perception in Rural China
School of Geography and Planning, Sun Yat-Sen University(中山大学地理科学与规划学院);China Regional Coordinated Development and Rural Construction Institute, Urbanization Institute, Sun Yat-Sen University(中山大学中国区域协调发展与乡村建设研究院、城市化研究院);Key Laboratory of Rural Digital Technology, Ministry of Housing and Urban-Rural Development(住房和城乡建设部乡村数字化技术重点实验室);School of Geography and Information Engineering, China University of Geosciences(中国地质大学地理与信息工程学院)
Tombo Propeller: Bio-Inspired Deformable Structure toward Collision-Accommodated Control for Drones
Soft Haptics Lab., School of Materials Science, Japan Advanced Institute of Science and Technology(日本先进科学技术学院材料科学学院软触觉实验室);Tandon School of Engineering, New York University(纽约大学坦登工程学院);Japan Science and Technology Agency, PRESTO(日本科学技术厅)
On Adversarial Attacks In Acoustic Drone Localization
Department of Computer Science(计算机科学系);Technion – Israel Institute of Technology(技术ion-以色列理工学院);School of Electrical and Computer Engineering(电气与计算机工程学院);Ben-Gurion University of the Negev(贝内尔谢瓦大学);Institute of Science and Technology Austria(奥地利科学与技术研究院)
FlyCo: Foundation Model-Empowered Drones for Autonomous 3D Structure Scanning in Open-World Environments
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology(香港科技大学电子与计算机工程系);School of Artificial Intelligence, Sun Yat-sen University(中山大学人工智能学院);Department of Mechanical and Energy Engineering, Southern University of Science and Technology(南方科技大学机械与能源工程系);Differential Robotics, Hangzhou, China(杭州差分机器人)
To Select or not to Select, that is the Question: Distilling Robot Skill Prediction into a Small Ensemble
Department of Computer Engineering and Software Engineering, Polytechnique Montréal(蒙特利尔理工学院计算机工程与软件工程系);MILA(蒙特利尔人工智能研究所);National Taiwan University of Science and Technology (NTUST)(台湾科技大学)