什么使得预训练视觉表示在鲁棒操作中成功？
What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?
AI总结：
本研究比较15个预训练视觉模型，发现涌现分割能力是预测操作任务分布外泛化的强指标，优于传统指标，并在模拟和真实环境验证。
摘要：
受计算机视觉中迁移学习成功的启发，机器人研究者们探索了视觉预训练作为提高从像素中学习的策略的学习效率和泛化能力的手段。为此，以往的工作倾向于使用大型物体交互数据集，例如人类完成多样化任务的第一人称视频，以追求与操作相关的特征。尽管这种方法提高了策略学习的效率，但在机器人应用中常见的分布偏移存在时，这些表示的可靠性仍不清楚。令人惊讶的是，我们发现为操作和控制任务设计的视觉表示在光照和场景纹理的细微变化或引入干扰物体时，并不一定能够泛化。为了理解哪些属性导致鲁棒的表示，我们比较了15个预训练视觉模型在不同视觉外观下的性能。我们发现，涌现的分割能力是ViT模型分布外泛化的强预测因子。由该指标产生的排名顺序比先前指导计算机视觉和机器学习泛化研究的指标（如下游ImageNet准确率、域内准确率或通过线索冲突性能评估的形状偏差）更具预测性。我们在两个模拟操作环境的十个任务中，对一系列分布偏移进行了广泛测试。在ALOHA设置上，分割分数能预测离线训练50个演示后的真实世界性能。
英文摘要：
Inspired by the success of transfer learning in computer vision, roboticists have investigated visual pre-training as a means to improve the learning efficiency and generalization ability of policies learned from pixels. To that end, past work has favored large object interaction datasets, such as first-person videos of humans completing diverse tasks, in pursuit of manipulation-relevant features. Although this approach improves the efficiency of policy learning, it remains unclear how reliable these representations are in the presence of distribution shifts that arise commonly in robotic applications. Surprisingly, we find that visual representations designed for manipulation and control tasks do not necessarily generalize under subtle changes in lighting and scene texture or the introduction of distractor objects. To understand what properties do lead to robust representations, we compare the performance of 15 pre-trained vision models under different visual appearances. We find that emergent segmentation ability is a strong predictor of out-of-distribution generalization among ViT models. The rank order induced by this metric is more predictive than metrics that have previously guided generalization research within computer vision and machine learning, such as downstream ImageNet accuracy, in-domain accuracy, or shape-bias as evaluated by cue-conflict performance. We test this finding extensively on a suite of distribution shifts in ten tasks across two simulated manipulation environments. On the ALOHA setup, segmentation score predicts real-world performance after offline training with 50 demonstrations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
