关于数据在开放集具身助益中的优势与劣势
On the Strengths and Weaknesses of Data for Open-set Embodied Assistance
AI总结:
本文研究了多模态基础模型在开放集辅助任务中的泛化能力,通过合成数据集评估模型在新用户行为和新配置中的表现,揭示了辅助数据集设计对模型性能的关键影响。
摘要:
具身基础模型在现实领域如机器人或自动驾驶中正变得越来越高效。这些模型通常部署在交互或辅助设置中,其中重要的是这些辅助模型能够泛化到新用户和新任务。多样化的交互数据生成为提供数据高效的泛化能力提供了有希望的途径。在本文中,我们研究了一个在合成领域上微调于多样化交互助益数据的多模态基础模型的泛化能力。我们沿着两个轴线探索泛化能力:a)协助未见过的用户行为类别;b)在训练中未遇到的新配置中提供指导。我们研究了一种称为"开放集纠正辅助"的广泛能力,其中模型需要检查长时间的用户行为并通过纠正行动或基于语言的反馈提供帮助。此任务在先前工作中仍未解决,通常假设封闭的纠正类别或依赖外部规划器,使其成为评估辅助数据限制的挑战性测试平台。为了支持此任务,我们生成了Overcooked中的合成辅助数据集,并微调了一个基于LLaMA的模型以评估对新任务和用户行为的泛化能力。我们的方法提供了对所需辅助数据集本质的关键见解。特别是,我们表明高性能模型受益于涵盖协助不同方面的数据集,包括多模态 grounding、缺陷推理和接触多样化场景。
英文摘要:
Embodied foundation models are increasingly performant in real-world domains such as robotics or autonomous driving. These models are often deployed in interactive or assistive settings, where it is important that these assistive models generalize to new users and new tasks. Diverse interactive data generation offers a promising avenue for providing data-efficient generalization capabilities for interactive embodied foundation models. In this paper, we investigate the generalization capabilities of a multimodal foundation model fine-tuned on diverse interactive assistance data in a synthetic domain. We explore generalization along two axes: a) assistance with unseen categories of user behavior and b) providing guidance in new configurations not encountered during training. We study a broad capability called \textbf{Open-Set Corrective Assistance}, in which the model needs to inspect lengthy user behavior and provide assistance through either corrective actions or language-based feedback. This task remains unsolved in prior work, which typically assumes closed corrective categories or relies on external planners, making it a challenging testbed for evaluating the limits of assistive data. To support this task, we generate synthetic assistive datasets in Overcooked and fine-tune a LLaMA-based model to evaluate generalization to novel tasks and user behaviors. Our approach provides key insights into the nature of assistive datasets required to enable open-set assistive intelligence. In particular, we show that performant models benefit from datasets that cover different aspects of assistance, including multimodal grounding, defect inference, and exposure to diverse scenarios.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。