Reward machines (RMs) are an effective approach for addressing non-Markovian rewards in reinforcement learning (RL) through finite-state machines. Traditional RMs, which label edges with propositional logic formulae, inherit the limited expressivity of propositional logic. This limitation hinders the learnability and transferability of RMs since complex tasks will require numerous states and edges. To overcome these challenges, we propose First-Order Reward Machines ($\texttt{FORM}$s), which use first-order logic to label edges, resulting in more compact and transferable RMs. We introduce a novel method for $\textbf{learning}$ $\texttt{FORM}$s and a multi-agent formulation for $\textbf{exploiting}$ them and facilitate their transferability, where multiple agents collaboratively learn policies for a shared $\texttt{FORM}$. Our experimental results demonstrate the scalability of $\texttt{FORM}$s with respect to traditional RMs. Specifically, we show that $\texttt{FORM}$s can be effectively learnt for tasks where traditional RM learning approaches fail. We also show significant improvements in learning speed and task transferability thanks to the multi-agent learning framework and the abstraction provided by the first-order language.
ConTraIRL: Factorized Contrastive Abstractions for Transferable IRL
School of Computing University of Georgia（乔治亚大学计算学院）；School of Computing Sciences & Computer Engineering The University of Southern Mississippi（密西西比大学计算科学与计算机工程学院）
Do It for HER: First-Order Temporal Logic Reward Specification in Reinforcement Learning (Extended Version)
DEIB, Politecnico di Milano（博洛尼亚工程学院，米兰理工大学）；Politecnico di Milano（米兰理工大学）；INESC-ID/Instituto Superior Técnico Universidade de Lisboa（里斯本大学超级学院/葡萄牙信息科学与技术研究所）；Dipartimento di Informatica Università degli Studi di Milano（米兰大学信息学院）
Planning with Abstract Learned Models While Learning Transferable Subtasks
University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）；Carnegie Mellon University（卡内基梅隆大学）；Georgia Institute of Technology（佐治亚理工学院）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Simmons University（西蒙斯大学）
Transferable Delay-Aware Reinforcement Learning via Implicit Causal Graph Modeling
College of Computer Science and Technology, National University of Defense Technology（计算机科学与技术学院，国防科技大学）；Intelligent Game and Decision Lab (IGDL)（智能游戏与决策实验室）；Institute of Military Transportation（军事交通运输研究院）；School of Artificial Intelligence, Hebei University of Technology（人工智能学院，河北工业大学）
Spook the Machine: Gamified Exploration of Human Imagination of Machine Fear
Max Planck Institute for Human Development（马克斯·普朗克人类发展研究所）；Max Planck Institute for Software Systems（马克斯·普朗克软件系统研究所）；Tallinn University（塔林大学）；Technische Universität Berlin（柏林工业大学）
本研究通过游戏化平台Spook the Machine探索人类对AI恐惧的反应，发现情感表达加深参与度，而奖励新颖性维持创作多样性，二者互补引导集体人机交互。
EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning
National Engineering Research Center of Software Engineering, Peking University（北京大学软件工程国家工程研究中心）；School of Computer Science, Peking University（北京大学计算机学院）；Key Laboratory of High Confidence Software Technologies, Ministry of Education（教育部高可信软件技术重点实验室）；Peking University Information Technology Institute, Tianjin Binhai（北京大学滨海信息技术研究院）；Research Institute, GRGBanking Equipment Co., Ltd.（广电运通金融电子股份有限公司研究院）
Exploratory Diffusion Model for Unsupervised Reinforcement Learning
Department of Computer Science & Technology, Institute for AI, BNRist Center, Tsinghua-Bosch Joint ML Center, THBI Lab, Tsinghua University（计算机科学与技术系、人工智能研究院、BNRist中心、清华-博世联合机器学习中心、THBI实验室、清华大学）
Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
IIIS, Tsinghua University, Beijing, China（清华大学信息科学技术学院，北京，中国）；Shanghai Qi Zhi Institute, Shanghai, China（上海启智研究院，上海，中国）；Department of Computer Science, University of California, Los Angeles, California, USA（计算机科学系，加州大学洛杉矶分校，美国，加州）
Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics
Department of Computer Science, Aalto University（艾尔沃斯大学计算机科学系）；Department of Computer Science, Toronto Metropolitan University（多伦多 Metropolitan 大学计算机科学系）；Department of Computer Science, University of Manchester（曼彻斯特大学计算机科学系）；Intel Corporation（英特尔公司）
Can Computational Reducibility Lead to Transferable Models for Graph Combinatorial Optimization?
Operations Research, Université de Montréal（运营研究，蒙特利尔大学）；Mila – Quebec AI Institute（魁北克人工智能研究所）；Valence Labs（Valence实验室）；Dept. of Mathematics, Boise State University（数学系，博伊州立大学）
