MineRL BASALT从人类反馈中学习比赛
The MineRL BASALT Competition on Learning from Human Feedback
摘要：
在过去的十年里，人们对深度学习研究的兴趣显著增加，许多公开的成功证明了它的潜力。因此，这些系统现在被纳入商业产品。随之而来的是一个额外的挑战：我们如何构建人工智能系统来解决没有明确定义的规范的任务？虽然已经提出了多种解决方案，但在这次比赛中，我们特别关注一个：从人类反馈中学习。我们不是使用预定义的奖励函数或使用具有预定义类别集的标记数据集来训练AI系统，而是使用从某种形式的人类反馈中获得的学习信号来训练AI系统，这些反馈可以随着对任务的理解的变化而演变，或者随着AI系统的能力的提高。MinerRL BASALT竞赛旨在推动对这一重要技术类别的研究。我们在Minecraft中设计了一套四个任务，我们预计很难写下硬编码的奖励函数。这些任务由一段自然语言定义：例如，“创建一个瀑布并为它拍摄一张风景照片”，并附带额外的澄清细节。参与者必须使用他们想要的任何方法为每个任务训练一个单独的代理。然后由阅读任务描述的人对代理进行评估。为了帮助参与者开始，我们提供了四个任务中每个任务的人类演示数据集，以及利用这些演示的模仿学习基线。我们希望，这项竞赛将提高我们构建人工智能系统的能力，使其能够按照设计者的意图去做，即使这种意图很难形式化。除了允许人工智能解决更多的任务外，这还可以更有效地监管人工智能系统，并在价值对齐问题上取得进展。
英文摘要：
The last decade has seen a significant increase of interest in deep learning research, with many public successes that have demonstrated its potential. As such, these systems are now being incorporated into commercial products. With this comes an additional challenge: how can we build AI systems that solve tasks where there is not a crisp, well-defined specification? While multiple solutions have been proposed, in this competition we focus on one in particular: learning from human feedback. Rather than training AI systems using a predefined reward function or using a labeled dataset with a predefined set of categories, we instead train the AI system using a learning signal derived from some form of human feedback, which can evolve over time as the understanding of the task changes, or as the capabilities of the AI system improve. The MineRL BASALT competition aims to spur forward research on this important class of techniques. We design a suite of four tasks in Minecraft for which we expect it will be hard to write down hardcoded reward functions. These tasks are defined by a paragraph of natural language: for example, "create a waterfall and take a scenic picture of it", with additional clarifying details. Participants must train a separate agent for each task, using any method they want. Agents are then evaluated by humans who have read the task description. To help participants get started, we provide a dataset of human demonstrations on each of the four tasks, as well as an imitation learning baseline that leverages these demonstrations. Our hope is that this competition will improve our ability to build AI systems that do what their designers intend them to do, even when the intent cannot be easily formalized. Besides allowing AI to solve more tasks, this can also enable more effective regulation of AI systems, as well as making progress on the value alignment problem.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
