arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.13467cs.ROcs.CV
收藏

CLIPSwarm:使用视觉-语言模型从文本提示生成无人机表演

CLIPSwarm: Generating Drone Shows from Text Prompts with Vision-Language Models

Instituto de Investigación en Ingeniería de Aragón, Universidad de Zaragoza(萨拉戈萨大学阿拉贡工程研究所);Stanford University(斯坦福大学)

Pablo Pueyo, Eduardo Montijano, Ana C. Murillo, Mac Schwager

AI总结:

CLIPSwarm提出一种基于自然语言自动化建模无人机编队的算法,利用CLIP评估文本与图像相似度,通过探索与利用迭代优化编队,并在逼真模拟中验证了其准确建模与无碰撞控制的有效性。

摘要:

本文介绍了CLIPSwarm,这是一种旨在基于自然语言自动化建模群体无人机编队的新算法。该算法首先丰富提供的单词,组成一个文本提示,作为迭代方法的输入,以寻找最匹配该单词的编队。算法采用“探索”和“利用”的不同步骤,迭代地优化机器人编队以与文本描述对齐。我们的框架目前在简单的编队目标上进行评估,仅限于轮廓形状。编队通过alpha-shape轮廓进行视觉表示,并自动为输入单词找到最具代表性的颜色。为了测量描述与编队视觉表示之间的相似度,我们使用CLIP[1],将文本和图像编码为向量并评估它们的相似度。随后,算法在可用无人机的给定约束下重新排列编队,以更有效地从视觉上表示该单词。然后为无人机分配控制动作,确保机器人行为和无碰撞移动。实验结果证明了该系统在从自然语言描述中准确建模机器人编队方面的有效性。通过在具有不同形状的逼真模拟中执行无人机表演,展示了该算法的通用性。我们建议读者参考补充视频以获取结果的可视化参考。

英文摘要:

This paper introduces CLIPSwarm, a new algorithm designed to automate the modeling of swarm drone formations based on natural language. The algorithm begins by enriching a provided word, to compose a text prompt that serves as input to an iterative approach to find the formation that best matches the provided word. The algorithm iteratively refines formations of robots to align with the textual description, employing different steps for "exploration" and "exploitation". Our framework is currently evaluated on simple formation targets, limited to contour shapes. A formation is visually represented through alpha-shape contours and the most representative color is automatically found for the input word. To measure the similarity between the description and the visual representation of the formation, we use CLIP [1], encoding text and images into vectors and assessing their similarity. Subsequently, the algorithm rearranges the formation to visually represent the word more effectively, within the given constraints of available drones. Control actions are then assigned to the drones, ensuring robotic behavior and collision-free movement. Experimental results demonstrate the system's efficacy in accurately modeling robot formations from natural language descriptions. The algorithm's versatility is showcased through the execution of drone shows in photorealistic simulation with varying shapes. We refer the reader to the supplementary video for a visual reference of the results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑