arXiv 2310.059052024-03-08cs.LGcs.AIcs.RO

TAIL：面向大预训练模型模仿学习的任务特定适配器

TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Models

Carnegie Mellon University（卡内基梅隆大学）；University of Southern California（南加利福尼亚大学）；Amazon Web Services（亚马逊网络服务）

Zuxin Liu, Jesse Zhang, Kavosh Asadi, Yao Liu, Ding Zhao, Shoham Sabach, Rasool Fakoor

备注：: Published on ICLR 2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对大预训练模型在控制领域数据稀缺、微调成本高的问题，提出TAIL框架，采用LoRA等参数高效微调技术实现有限数据下的任务适配，仅用1%全微调参数就达到最优性能且避免灾难性遗忘。

摘要：

大预训练模型在机器人等控制领域的全部潜力仍远未被挖掘，这主要是因为此类应用中数据稀缺，且训练或微调这些大模型存在计算挑战。现有研究主要侧重于针对决策任务的大模型高效预训练，或是单任务适配，但现实问题要求针对新控制任务实现数据高效的持续适配。针对这些限制，我们提出TAIL（Task-specific Adapters for Imitation Learning，面向模仿学习的任务特定适配器），这是一个用于高效适配新控制任务的框架。受语言领域参数高效微调的最新进展启发，我们在TAIL中探索了瓶颈适配器（Bottleneck Adapters）、P-Tuning、低秩适配（Low-Rank Adaptation, LoRA）等高效微调技术，以在演示数据有限的情况下让大预训练模型适配新任务。我们在大规模语言条件操控任务中开展了大量实验，对比了主流的参数高效微调技术与适配基线，结果表明搭载LoRA的TAIL仅需全微调1%的可训练参数就能达到最佳适配后性能，同时在持续学习场景下能避免灾难性遗忘并保留适配可塑性。

英文摘要：

The full potential of large pretrained models remains largely untapped in control domains like robotics. This is mainly because of the scarcity of data and the computational challenges associated with training or fine-tuning these large models for such applications. Prior work mainly emphasizes either effective pretraining of large models for decision-making or single-task adaptation. But real-world problems will require data-efficient, continual adaptation for new control tasks. Recognizing these constraints, we introduce TAIL (Task-specific Adapters for Imitation Learning), a framework for efficient adaptation to new control tasks. Inspired by recent advancements in parameter-efficient fine-tuning in language domains, we explore efficient fine-tuning techniques -- e.g., Bottleneck Adapters, P-Tuning, and Low-Rank Adaptation (LoRA) -- in TAIL to adapt large pretrained models for new tasks with limited demonstration data. Our extensive experiments in large-scale language-conditioned manipulation tasks comparing prevalent parameter-efficient fine-tuning techniques and adaptation baselines suggest that TAIL with LoRA can achieve the best post-adaptation performance with only 1\% of the trainable parameters of full fine-tuning, while avoiding catastrophic forgetting and preserving adaptation plasticity in continual learning settings.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。