Post-training of frontier language models is conducted on curated task suites, and inevitably leaves a distribution shift between training and deployment environments. This exposes developers to generalization failures, which are relatively poorly understood. To better understand such generalization failures, we believe the community should construct clean demonstrations under simplified conditions. To facilitate this, we propose a simple and flexible way to construct language models which fail to generalize in controllable ways when subsequently trained with Reinforcement Learning (RL) on a given distribution of training tasks. Our construction uses Supervised Fine-Tuning on a dataset of a mixture of transcripts corresponding to a collection of 'conditional policies', which can each independently be assigned certain behaviors on each different task distribution, to obtain a model that is then well approximated as a 'mixture of conditional policies.' We observe that RL training then selects for policies that obtain the highest reward on the training distribution. This can produce striking behaviors: in a controlled setting with two distributions containing identical questions prepended with two different 'trigger strings', RL training on either distribution actively degrades performance on the other to zero, even though the underlying task is identical. We also use our construction to illustrate two novel ways in which generalization may fail in future language models, corresponding to distribution shifts of task coverage and temporal context respectively. While our construction is deliberately simple and may not closely resemble 'natural' generalization failures, the resulting 'model organisms' are of interest for alignment stress-testing and generalization science, and can be used as existence proofs that training success and generalization can come apart in structured ways.
Masaryk University（马萨里克大学）；Institute of Science and Technology Austria（奥地利科学技术研究所）
Markov decision processes (MDPs) are the defacto frame-work for sequential decision making in the presence ofstochastic uncertainty. A classical optimization criterion forMDPs is to maximize the expected discounted-sum pay-off, which ignores low probability catastrophic events withhighly negative impact on the system. On the other hand,risk-averse policies require the probability of undesirableevents to be below a given threshold, but they do not accountfor optimization of the expected payoff. We consider MDPswith discounted-sum payoff with failure states which repre-sent catastrophic outcomes…
Is Conditional Generative Modeling all you need for Decision-Making?
Massachusetts Institute of Technology（麻省理工学院）；Improbable AI Lab（Improbable AI实验室）；Operations Research Center（运筹学中心）；Computer Science and Artificial Intelligence Lab（计算机科学与人工智能实验室）
We introduce a distributional method for learning the optimal policy in risk averse Markov decision process with finite state action spaces, latent costs, and stationary dynamics. We assume sequential observations of states, actions, and costs and assess the performance of a policy using dynamic risk measures constructed from nested Kusuoka-type conditional risk mappings. For such performance criteria, randomized policies may outperform deterministic policies, therefore, the candidate policies lie in the d-dimensional simplex where d is the cardinality of the action space. Existing risk averse…
University of Science and Technology of China（中国科学技术大学）；SUNY Binghamton（纽约州立大学宾汉姆顿分校）
Reinforcement learning methods have been used for learning dialogue policies. However, learning an effective dialogue policy frequently requires prohibitively many conversations. This is partly because of the sparse rewards in dialogues, and the very few successful dialogues in early learning phase. Hindsight experience replay (HER) enables learning from failures, but the vanilla HER is inapplicable to dialogue learning due to the implicit goals. In this work, we develop two complex HER methods providing different trade-offs between complexity and performance, and, for the first time, enabled …
Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent
Laboratory for Big Data and Decision（大数据与决策实验室）；National University of Defense Technology（国防科技大学）；Samsung AI Center Cambridge（三星AI研究中心）；Queen Mary University of London（伦敦玛丽女王大学）；Fudan University（复旦大学）；ShanghaiTech University（上海科技大学）；Shanghai Innovation Institute（上海创新研究院）
Humans decompose novel complex tasks into simpler ones to exploit previously learned skills. Analogously, hierarchical reinforcement learning seeks to leverage lower-level policies for simple tasks to solve complex ones. However, because each lower-level policy induces a different distribution of states, transitioning from one lower-level policy to another may fail due to an unexpected starting state. We introduce transition policies that smoothly connect lower-level policies by producing a distribution of states and actions that matches what is expected by the next policy. Training transition…
The AUTOLab at University of California, Berkeley（加州大学伯克利分校AUTOLab实验室）
A new generation of automated bin picking systems using deep learning is evolving to support increasing demand for e-commerce. To accommodate a wide variety of products, many automated systems include multiple gripper types and/or tool changers. However, for some objects, sequential grasp failures are common: when a computed grasp fails to lift and remove the object, the bin is often left unchanged; as the sensor input is consistent, the system retries the same grasp over and over, resulting in a significant reduction in mean successful picks per hour (MPPH). Based on an empirical study of seq…
ActProbe: Action-Space Probe for Early Failure Detection of Generative Robot Policies
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；University of Electronic Science and Technology of China（电子科技大学）；Nanjing University（南京大学）
Failure Prediction at Runtime for Generative Robot Policies
Technical University of Munich（慕尼黑工业大学）；Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与机器智能研究院）；Robotics Institute Germany（德国机器人研究所）；Munich Center for Machine Learning（慕尼黑机器学习中心）
Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
Multisensory polices are known to enhance both state estimation and target tracking. However, in the space of end-to-end sensorimotor control, this multi-sensor outlook has received limited attention. Moreover, systematic ways to make policies robust to partial sensor failure are not well explored. In this work, we propose a specific customization of Dropout, called \textit{Sensor Dropout}, to improve multisensory policy robustness and handle partial failure in the sensor-set. We also introduce an additional auxiliary loss on the policy network in order to reduce variance in the band of potent…
Moving On, Even When You're Broken: Fail-Active Trajectory Generation via Diffusion Policies Conditioned on Embodiment and Task
Dept. of Computer Science, University of Colorado Boulder, CO, USA（计算机科学系，科罗拉多大学博尔德分校）；Dept. of Electrical, Computer, and Energy Engineering, University of Colorado Boulder, CO, USA（电气、计算机与能源工程系，科罗拉多大学博尔德分校）
Rewind-IL: Online Failure Detection and State Respawning for Imitation Learning
College of Connected Computing, Vanderbilt University（范德比尔特大学连接计算学院）；School of Computer Science, University of Waterloo（滑铁卢大学计算机科学学院）；School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；Australian Centre for Robotics, The University of Sydney（悉尼大学机器人研究中心）
Learning to play optimally against any mixture over a diverse set of strategies is of important practical interests in competitive games. In this paper, we propose simplex-NeuPL that satisfies two desiderata simultaneously: i) learning a population of strategically diverse basis policies, represented by a single conditional network; ii) using the same network, learn best-responses to any mixture over the simplex of basis policies. We show that the resulting conditional policies incorporate prior information about their opponents effectively, enabling near optimal returns against arbitrary mixt…
