How does Your RL Agent Explore? An Optimal Transport Analysis of Occupancy Measure Trajectories
The University of Sheffield（谢菲尔德大学）；Univ. Lille（里尔大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Centrale Lille（里尔中央理工学院）；Centrum Wiskunde & Informatica（数学与计算机科学中心）
Reabetswe M. Nkhumise, Debabrota Basu, Tony J. Prescott, Aditya Gilra
The rising successes of RL are propelled by combining smart algorithmic strategies and deep architectures to optimize the distribution of returns and visitations over the state-action space. A quantitative framework to compare the learning processes of these eclectic RL algorithms is currently absent but desired in practice. We address this gap by representing the learning process of an RL algorithm as a sequence of policies generated during training, and then studying the policy trajectory induced in the manifold of state-action occupancy measures. Using an optimal transport-based metric, we measure the length of the paths induced by the policy sequence yielded by an RL algorithm between an initial policy and a final optimal policy. Hence, we first define the 'Effort of Sequential Learning' (ESL). ESL quantifies the relative distance that an RL algorithm travels compared to the shortest path from the initial to the optimal policy. Further, we connect the dynamics of policies in the occupancy measure space and regret (another metric to understand the suboptimality of an RL algorithm), by defining the 'Optimal Movement Ratio' (OMR). OMR assesses the fraction of movements in the occupancy measure space that effectively reduce an analogue of regret. Finally, we derive approximation guarantees to estimate ESL and OMR with finite number of samples and without access to an optimal policy. Through empirical analyses across various environments and algorithms, we demonstrate that ESL and OMR provide insights into the exploration processes of RL algorithms and hardness of different tasks in discrete and continuous MDPs.
TrajFlow: A Generative Framework for Occupancy Density Estimation Using Normalizing Flows
Department of Computer Science University of Minnesota（计算机科学系明尼苏达大学）；Department of Civil Environmental and Geo- Engineering University of Minnesota（土木环境与地球工程系明尼苏达大学）
Comparative Analysis of Polygon-Based and Global Machine Learning Models for Bus Occupancy Prediction
Faculty of Computer and Information Science, University of Ben Gurion, Be'er Sheva, Israel（本·古里安大学计算机与信息科学学院，贝尔谢巴，以色列）；Department of Geography and Environmental Development, Ben-Gurion, Israel（地理与环境发展系，本·古里安，以色列）
TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination
Department of Electrical \& Computer Engineering, University of Arizona；Engineering College, Soochow University；INSAIT, Sofia University "St. Kliment Ohridski"；Department of Electrical；Computer Engineering, Stony Brook University
