The definition and representation of planning problems is at the heart of AI planning research. A key part is the representation of action models. Decades of advances improving declarative action model representations resulted in numerous theoretical advances, and capable, working, domain-independent planners. However, despite the maturity of the field, AI planning technology is still rarely used outside the research community, suggesting that current representations fail to capture real-world requirements, such as utilizing complex mathematical functions and models learned from data. We argue that this is because the modeling process is assumed to have taken place and completed prior to the planning process, i.e., offline modeling for offline planning. There are several challenges inherent to this approach, including: limited expressiveness of declarative modeling languages; early commitment to modeling choices and computation, that preclude using the most appropriate resolution for each action model -- which can only be known during planning; and difficulty in reliably using non-declarative, learned, models.
We therefore suggest to change the AI planning process, such that is carries out online modeling in offline planning, i.e., the use of action models that are computed or even generated as part of the planning process, as they are accessed. This generalizes the existing approach (offline modeling). The proposed definition admits novel planning processes, and we suggest one concrete implementation, demonstrating the approach. We sketch initial results that were obtained as part of a first attempt to follow this approach by planning with action cost estimators. We conclude by discussing open challenges.
Combining Offline Models and Online Monte-Carlo Tree Search for Planning from Scratch
Xiamen University（厦门大学）
随机和部分可观测环境中的规划问题是人工智能的一个核心问题。解决这类问题的一个常用技术是首先构建一个精确的模型。虽然最近已经提出了一些方法来学习模型不确定性下的最佳行为，先验知识的环境仍然需要保证所提出的算法的性能。利用预测状态表示（Predictive State Representations~（PSR））方法的状态表示和模型预测的优点，本文提出了一种从头开始规划的方法，该方法首先离线学习PSR模型，然后结合在线蒙特卡罗树搜索进行模型不确定性规划。通过与模型不确定性规划方法的比较，证明了所提出方法的有效性及其收敛性。我们所提出的方法的有效性和可扩展性也进行了测试的RockSample问题，这是不可行的最先进的BA-POMDP为基础的方法。
Model-Based Offline Planning with Trajectory Pruning
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；JD iCity & JD Intelligent Cities Research, JD Technology（京东科技京东城市与京东智能城市研究院）
Department of Mechanical Engineering, Iowa State University（伊阿华州立大学机械工程系）；Department of Computer Science, Iowa State University（伊阿华州立大学计算机科学系）；Department of Industrial and Manufacturing Systems Engineering, Iowa State University（伊阿华州立大学工业与制造系统工程系）；Translational AI Center, Iowa State University（伊阿华州立大学转化人工智能中心）
Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learning
Harbin Institute of Technology（哈尔滨工业大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Northwestern Polytechnical University（西北工业大学）；Shenzhen Research Institute of Northwestern Polytechnical University（西北工业大学深圳研究院）
GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models
University of Warwick（华威大学）；Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）；University of Cambridge（剑桥大学）；University of Liverpool（利物浦大学）
Sim2O: Efficient Offline-to-Online MARL via Joint Action Composition
Department of Computer Science and Engineering, Washington University in St. Louis（圣路易斯华盛顿大学计算机科学与工程系）；The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China（中国科学院自动化研究所复杂系统认知与决策智能重点实验室）
A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
The University of Sydney（悉尼大学）；SenseTime Computer Vision Group（商汤科技计算机视觉组）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；CUHK MMLab（香港中文大学多媒体实验室）；Institute for AI, Peking University（北京大学人工智能研究院）
Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing Innovation Center of Humanoid Robotics（北京人形机器人创新中心）；Beijing Forestry University（北京林业大学）；Peking University（北京大学）；Microsoft Research（微软研究院）
A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE（知识驱动的人机智能工程研究中心，教育部）；International Center of Future Science, Jilin University（未来科学国际中心，吉林大学）；JPMorgan AI Research（摩根大通AI研究）；Northwestern Polytechnical University（西北工业大学）；Lehigh University（莱斯大学）
When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning
Tsinghua University（清华大学）；Indian Institute of Technology（印度理工学院）；Shanghai Jiaotong University（上海交通大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Shanghai AI Laboratory（上海人工智能实验室）
