上下文决策Transformer：通过层次化思维链进行强化学习
In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought
AI总结：
针对上下文强化学习中序列过长导致计算成本高的问题，提出上下文决策Transformer（IDT），通过引入人类决策的层次结构重构序列，利用高层决策指导多步低层动作，在长步长任务中实现高效自我改进并大幅提升在线评估速度。
摘要：
上下文学习是离线强化学习（RL）处理在线任务的一种有前景的方法，可通过提供任务提示来实现。近期研究表明，当将RL任务视为跨回合的序列预测问题时，上下文RL能够以试错的方式实现自我改进。尽管这种自我改进不需要梯度更新，但随着跨回合序列随任务步长的增加，当前方法仍面临高昂的计算成本。为此，我们提出了一种上下文决策Transformer（IDT），以高层试错的方式实现自我改进。具体而言，IDT受人类决策高效层次结构的启发，将序列重构为由高层决策组成，而非与环境交互的低层动作。由于一个高层决策可指导多步低层动作，IDT自然避免了序列过长，更高效地解决在线任务。实验结果表明，在长步长任务上，IDT超越了当前的上下文RL方法，达到了最先进的性能。特别是在D4RL基准上，IDT的在线评估速度比基线快36倍，在Grid World基准上快27倍。
英文摘要：
In-context learning is a promising approach for offline reinforcement learning (RL) to handle online tasks, which can be achieved by providing task prompts. Recent works demonstrated that in-context RL could emerge with self-improvement in a trial-and-error manner when treating RL tasks as an across-episodic sequential prediction problem. Despite the self-improvement not requiring gradient updates, current works still suffer from high computational costs when the across-episodic sequence increases with task horizons. To this end, we propose an In-context Decision Transformer (IDT) to achieve self-improvement in a high-level trial-and-error manner. Specifically, IDT is inspired by the efficient hierarchical structure of human decision-making and thus reconstructs the sequence to consist of high-level decisions instead of low-level actions that interact with environments. As one high-level decision can guide multi-step low-level actions, IDT naturally avoids excessively long sequences and solves online tasks more efficiently. Experimental results show that IDT achieves state-of-the-art in long-horizon tasks over current in-context RL methods. In particular, the online evaluation time of our IDT is \textbf{36$\times$} times faster than baselines in the D4RL benchmark and \textbf{27$\times$} times faster in the Grid World benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
