系统下一步应该做什么?:用于估计系统动作的操作动作字幕
What Should the System Do Next?: Operative Action Captioning for Estimating System Actions
摘要:
像机器人这样的人类辅助系统需要根据观察正确地理解周围的情况,并为人类输出所需的支持行动。语言是与人类交流的重要渠道之一,要求机器人具有表达其理解和行动规划结果的能力。在这项研究中,我们提出了一个新的任务的操作动作字幕,估计和verbalizes的系统在人类协助域所采取的行动。我们构建了一个系统,输出一个可能的操作动作,改变当前状态到给定的目标状态的口头描述。我们收集了一个数据集,包括两个图像作为观察,表示当前状态和通过动作改变的状态,以及描述将当前状态改变为目标状态的动作的说明,通过在日常生活中的众包。然后,我们构建了一个系统,估计操作动作的标题。由于操作动作的标题预计将包含一些状态变化的动作,我们使用场景图预测作为辅助任务,因为场景图中写入的事件对应于状态变化。实验结果表明,我们的系统成功地描述了当前和目标状态之间应该进行的操作动作。预测场景图的辅助任务提高了估计结果的质量。
英文摘要:
Such human-assisting systems as robots need to correctly understand the surrounding situation based on observations and output the required support actions for humans. Language is one of the important channels to communicate with humans, and the robots are required to have the ability to express their understanding and action planning results. In this study, we propose a new task of operative action captioning that estimates and verbalizes the actions to be taken by the system in a human-assisting domain. We constructed a system that outputs a verbal description of a possible operative action that changes the current state to the given target state. We collected a dataset consisting of two images as observations, which express the current state and the state changed by actions, and a caption that describes the actions that change the current state to the target state, by crowdsourcing in daily life situations. Then we constructed a system that estimates operative action by a caption. Since the operative action's caption is expected to contain some state-changing actions, we use scene-graph prediction as an auxiliary task because the events written in the scene graphs correspond to the state changes. Experimental results showed that our system successfully described the operative actions that should be conducted between the current and target states. The auxiliary tasks that predict the scene graphs improved the quality of the estimation results.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。