arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1707.04677cs.CV
收藏

面向任务动作预测的知识引导循环神经网络学习

Knowledge-Guided Recurrent Neural Network Learning for Task-Oriented Action Prediction

Sun Yat-Sen University(中山大学)

Liang Lin, Lili Huang, Tianshui Chen, Yukang Gan, Hui Cheng

摘要:

本文研究面向任务的动作预测,即,预测在特定场景下完成特定任务的动作序列,是计算机视觉研究中的一个新问题。主要的挑战在于如何建模特定于任务的知识,并将其整合到学习过程中。在这项工作中,我们提出训练一个循环的长短期记忆(LSTM)网络来处理这个问题,即,将场景图像(包括预先定位的对象)和指定的任务作为输入,并循环地预测动作序列。然而,训练这样的网络通常需要大量的注释样本来覆盖语义空间(例如,不同的动作分解和排序)。为了缓解这个问题,我们引入了一个时间与或图(AOG)的任务描述,分层表示成原子动作的任务。有了这个AOG表示,我们可以产生许多有效的样本(即,动作序列符合常识),用一小部分带注释的样本训练另一个辅助LSTM网络。并且这些生成的样本(即,面向任务的动作序列)有效地促进训练用于面向任务的动作预测的模型。在实验中,我们创建了一个包含不同日常任务的新数据集,并广泛评估了我们方法的有效性。

英文摘要:

This paper aims at task-oriented action prediction, i.e., predicting a sequence of actions towards accomplishing a specific task under a certain scene, which is a new problem in computer vision research. The main challenges lie in how to model task-specific knowledge and integrate it in the learning procedure. In this work, we propose to train a recurrent long-short term memory (LSTM) network for handling this problem, i.e., taking a scene image (including pre-located objects) and the specified task as input and recurrently predicting action sequences. However, training such a network usually requires large amounts of annotated samples for covering the semantic space (e.g., diverse action decomposition and ordering). To alleviate this issue, we introduce a temporal And-Or graph (AOG) for task description, which hierarchically represents a task into atomic actions. With this AOG representation, we can produce many valid samples (i.e., action sequences according with common sense) by training another auxiliary LSTM network with a small set of annotated samples. And these generated samples (i.e., task-oriented action sequences) effectively facilitate training the model for task-oriented action prediction. In the experiments, we create a new dataset containing diverse daily tasks and extensively evaluate the effectiveness of our approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑