arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.12093cs.ROcs.CV
收藏

面向家庭场景整理的LLM增强场景图学习

LLM-enhanced Scene Graph Learning for Household Rearrangement

National University of Defense Technology(国防科技大学);Shenzhen University(深圳大学)

Wenhao Li, Zhiyuan Yu, Qijin She, Zhinan Yu, Yuqing Lan, Chenyang Zhu, Ruizhen Hu, Kai Xu

AI总结:

针对家庭场景整理依赖常识与用户偏好的需求,提出LLM增强场景图学习方法,构建功能可供性增强图实现错位物品检测与归置规划,在自建基准上达到最优性能。

摘要:

家庭场景整理任务涉及识别场景中错位的物品并将其归置到合适的位置,该任务既依赖客观层面的常识知识,也依赖主观层面的人类用户偏好。为完成该任务,我们提出直接从场景本身挖掘与用户偏好对齐的物品功能,无需依赖人工干预。为此,我们采用场景图表示方法,提出LLM增强场景图学习框架,该框架将输入场景图转换为功能可供性增强图(affordance-enhanced graph,AEG),其中节点得到信息增强,且会发现新的边(关系)。在AEG中,对应容器类物品的节点会补充上下文诱导的功能可供性,该属性编码了可放置在其上的可携带物品类型;同时通过新发现的非局部关系生成新的边。基于AEG,我们通过检测错位的可携带物品并为每个物品确定合适的放置位置,执行场景整理的任务规划。我们通过在模拟器中搭建一个整理机器人来测试所提方法,并在我们构建的新基准上开展评估。大量评估结果表明,我们的方法在错位检测及后续的整理规划任务上均达到了state-of-the-art(当前最优)性能。

英文摘要:

The household rearrangement task involves spotting misplaced objects in a scene and accommodate them with proper places. It depends both on common-sense knowledge on the objective side and human user preference on the subjective side. In achieving such task, we propose to mine object functionality with user preference alignment directly from the scene itself, without relying on human intervention. To do so, we work with scene graph representation and propose LLM-enhanced scene graph learning which transforms the input scene graph into an affordance-enhanced graph (AEG) with information-enhanced nodes and newly discovered edges (relations). In AEG, the nodes corresponding to the receptacle objects are augmented with context-induced affordance which encodes what kind of carriable objects can be placed on it. New edges are discovered with newly discovered non-local relations. With AEG, we perform task planning for scene rearrangement by detecting misplaced carriables and determining a proper placement for each of them. We test our method by implementing a tiding robot in simulator and perform evaluation on a new benchmark we build. Extensive evaluations demonstrate that our method achieves state-of-the-art performance on misplacement detection and the following rearrangement planning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑