面向离线交互数据的距离加权监督学习
Distance Weighted Supervised Learning for Offline Interaction Data
AI总结：
针对序列决策中离线交互数据利用难题，提出距离加权监督学习DWSL方法，通过监督学习建模状态时间步分布近似最短路径提取策略，理论上可收敛至数据约束下的最优策略，实验中在多数据集尤其是高维图像领域性能优于现有目标条件IL和RL算法。
摘要：
序列决策算法通常难以有效利用不同来源的非结构化离线交互数据。基于监督学习的模仿学习（IL）方法鲁棒性强，但需要难以采集的最优演示数据。离线目标条件强化学习（RL）算法有望从次优数据中学习，但面临优化挑战，在高维数据场景下尤为突出。为填补IL与RL之间的研究空白，我们提出距离加权监督学习（Distance Weighted Supervised Learning，简称DWSL），这是一种从离线数据中学习目标条件策略的监督式方法。DWSL仅通过监督学习对离线数据中状态之间的所有时间步分布进行建模，并利用该分布近似最短路径距离。为提取策略，我们根据动作对距离估计值的缩减程度为其赋予权重。理论上，DWSL无需任何自举操作，即可收敛到受数据分布约束的最优策略，这是离线学习中极具吸引力的特性。在我们测试的所有数据集上，实验结果表明DWSL以行为克隆性能为下限，同时仍能实现策略提升。在高维图像领域，DWSL的性能超越了此前的目标条件IL与RL算法。可视化结果与代码可访问https://sites.google.com/view/dwsl/home 查看。
英文摘要：
Sequential decision making algorithms often struggle to leverage different sources of unstructured offline interaction data. Imitation learning (IL) methods based on supervised learning are robust, but require optimal demonstrations, which are hard to collect. Offline goal-conditioned reinforcement learning (RL) algorithms promise to learn from sub-optimal data, but face optimization challenges especially with high-dimensional data. To bridge the gap between IL and RL, we introduce Distance Weighted Supervised Learning or DWSL, a supervised method for learning goal-conditioned policies from offline data. DWSL models the entire distribution of time-steps between states in offline data with only supervised learning, and uses this distribution to approximate shortest path distances. To extract a policy, we weight actions by their reduction in distance estimates. Theoretically, DWSL converges to an optimal policy constrained to the data distribution, an attractive property for offline learning, without any bootstrapping. Across all datasets we test, DWSL empirically maintains behavior cloning as a lower bound while still exhibiting policy improvement. In high-dimensional image domains, DWSL surpasses the performance of both prior goal-conditioned IL and RL algorithms. Visualizations and code can be found at https://sites.google.com/view/dwsl/home .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
