用于评估语言引导机器人策略的对比集
Contrast Sets for Evaluating Language-Guided Robot Policies
AI总结：
本文提出用于语言引导机器人策略评估的对比集，通过对i.i.d.测试实例作针对性扰动，以较低实验成本揭示策略性能变化，并在模拟操作和实体机器人VLN任务中验证其价值。
摘要：
在语言引导的真实世界场景中，机器人评估耗时，且通常只在复杂场景中采样潜在指令空间的一小部分。在本工作中，我们将机器人领域的对比集作为一种方法引入，用于对原本独立同分布（i.i.d.）的测试实例进行微小但有针对性的扰动。我们研究了实验者执行评估所需投入与所得测试性能估计之间的关系，以及可从受扰动实例上的性能中获得的洞见。我们在一个模拟操作任务和一个实体机器人视觉与语言导航任务中，利用不同对比集扰动带来的相对性能变化，以更低的实验者投入来刻画策略。我们鼓励使用对比集评估，将其作为实体机器人上小规模、i.i.d.演示的更具信息量的替代方案，以及工业规模真实世界评估的可扩展替代方案。
英文摘要：
Robot evaluations in language-guided, real world settings are time-consuming and often sample only a small space of potential instructions across complex scenes. In this work, we introduce contrast sets for robotics as an approach to make small, but specific, perturbations to otherwise independent, identically distributed (i.i.d.) test instances. We investigate the relationship between experimenter effort to carry out an evaluation and the resulting estimated test performance as well as the insights that can be drawn from performance on perturbed instances. We use the relative performance change of different contrast set perturbations to characterize policies at reduced experimenter effort in both a simulated manipulation task and a physical robot vision-and-language navigation task. We encourage the use of contrast set evaluations as a more informative alternative to small scale, i.i.d. demonstrations on physical robots, and as a scalable alternative to industry-scale real world evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
