Prompt2Craft: 基于LLM生成功能性的手工艺装配
Prompt2Craft: Generating Functional Craft Assemblies with LLMs
AI总结：
Prompt2Craft利用LLM生成手工艺装配，通过识别可见部分、检索模板网格并优化姿态，最终生成功能性的手工艺作品。
摘要：
受传统手工艺启发，其中一个人根据可用物品即兴组装物体，我们正式引入了手工艺装配任务。这是一个机器人装配任务，旨在利用可用物品构建目标物体的准确表示，这些物品并不直接对应其组成部分。在本工作中，我们关注在给定输入为野外目标RGB图像时，选择最终手工艺中使用的物品子集。我们使用掩码分割神经网络来识别可见部分，随后检索带有标签的模板网格。这些网格经过姿态优化以确定最合适的模板。然后，我们提出将转换后的模板网格的部件简化为立方体或圆柱等基本形状。最后，我们设计了一种搜索算法，根据局部和全局比例在场景中查找对应关系。我们开发了用于比较的基线，考虑了所有可能的组合，并选择在前景图和掩码准确性中常用的指标中得分最高的组合。我们的方法在两个不同的场景中实现了与基线相当的结果，并展示了在现实场景中的实现定性结果。
英文摘要：
Inspired by traditional handmade crafts, where a person improvises assemblies based on the available objects, we formally introduce the Craft Assembly Task. It is a robotic assembly task that involves building an accurate representation of a given target object using the available objects, which do not directly correspond to its parts. In this work, we focus on selecting the subset of available objects for the final craft, when the given input is an RGB image of the target in the wild. We use a mask segmentation neural network to identify visible parts, followed by retrieving labeled template meshes. These meshes undergo pose optimization to determine the most suitable template. Then, we propose to simplify the parts of the transformed template mesh to primitive shapes like cuboids or cylinders. Finally, we design a search algorithm to find correspondences in the scene based on local and global proportions. We develop baselines for comparison that consider all possible combinations, and choose the highest scoring combination for common metrics used in foreground maps and mask accuracy. Our approach achieves comparable results to the baselines for two different scenes, and we show qualitative results for an implementation in a real-world scenario.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
