The emergence of compositional reasoning in large language models through reinforcement learning with verifiable rewards (RLVR) has been a key driver of recent empirical successes. Despite this progress, it remains unclear which compositional problems are learnable in this setting using outcome-level feedback alone. In this work, we theoretically study the learnability of compositional problems in autoregressive models under RLVR training. We identify a quantity that we call the task-advantage ratio, a joint property of the compositional problem and the base model, that characterizes which tasks and compositions are learnable from outcome-level feedback. On the positive side, using this characterization, we show that compositional problems where correct intermediate steps provide a clear advantage are efficiently learnable with RLVR. We also analyze how such an advantage naturally arises in different problems. On the negative side, when the structural advantage is not present, RLVR may converge to suboptimal compositions. We prove that, in some cases, the quality of the base model determines if such an advantage exists and whether RLVR will converge to a suboptimal solution. We hope our analysis can provide a principled theoretical understanding of when and why RLVR succeeds and when it does not.
DéjàQ: Open-Ended Evolution of Diverse, Learnable and Verifiable Problems
Willem Röpke AI Lab, Vrije Universiteit Brussel Belgium（维尔姆·罗普克人工智能实验室，布鲁塞尔自由大学）；FLAIR, University of Oxford United Kingdom（FLAIR，牛津大学）；University College London United Kingdom（伦敦大学学院）
Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin（得克萨斯大学奥斯汀分校奥登计算工程与科学研究所）；Uppsala University（乌普萨拉大学）；The University of Texas at Austin（得克萨斯大学奥斯汀分校）
We propose a framework for verifiable and compositional reinforcement learning (RL) in which a collection of RL subsystems, each of which learns to accomplish a separate subtask, are composed to achieve an overall task. The framework consists of a high-level model, represented as a parametric Markov decision process (pMDP) which is used to plan and to analyze compositions of subsystems, and of the collection of low-level subsystems themselves. By defining interfaces between the subsystems, the framework enables automatic decompositions of task specifications, e.g., reach a target set of states…
The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning
TMLR Group, Department of Computer Science, Hong Kong Baptist University（香港 Baptist 大学计算机科学系 TMLR 组）；Stanford University（斯坦福大学）；Sydney AI Centre, The University of Sydney（悉尼大学人工智能中心）
BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoning
Department of Mechanical Engineering（机械工程系）；Massachusetts Institute of Technology（麻省理工学院）；Department of Civil and Environmental Engineering（土木与环境工程系）；Schwarzman College of Computing（施瓦茨曼计算学院）
Compositional Policy Learning in Stochastic Control Systems with Formal Guarantees
Institute of Science and Technology Austria (ISTA)（奥地利科学技术研究所（ISTA））；Massachusetts Institute of Technology（麻省理工学院）；The Pennsylvania State University（宾夕法尼亚州立大学）
Trustless Federated Learning at Edge-Scale: A Compositional Architecture for Decentralized, Verifiable, and Incentive-Aligned Coordination
Marshall School of Business University of Southern California（南加州大学马歇尔商学院）；School of Natural and Computing Sciences University of Aberdeen（阿伯丁大学自然与计算科学学院）
Hybrid systems are a compact and natural mechanism with which to address problems in robotics. This work introduces an approach to learning hybrid systems from demonstrations, with an emphasis on extracting models that are explicitly verifiable and easily interpreted by robot operators. We fit a sequence of controllers using sequential importance sampling under a generative switching proportional controller task model. Here, we parameterise controllers using a proportional gain and a visually verifiable joint angle goal. Inference under this model is challenging, but we address this by introdu…
Agentic Proposing: Enhancing Large Language Model Reasoning via Compositional Skill Synthesis
AI DATA, Alibaba Group Holding Limited（阿里数据，阿里巴巴集团控股有限公司）；EPIC Lab, Shanghai Jiao Tong University（上海交通大学EPIC实验室）；Shanghai University of Finance（上海财经大学）；Wuhan University（武汉大学）
LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignment
Nanjing University（南京大学）；The Hong Kong Polytechnic University（香港理工大学）；The Chinese University of Hong Kong（香港中文大学）；University of Science and Technology of China（中国科学技术大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）
Data augmentation aims to enrich training samples for alleviating the overfitting issue in low-resource or class-imbalanced situations. Traditional methods first devise task-specific operations such as Synonym Substitute, then preset the corresponding parameters such as the substitution rate artificially, which require a lot of prior knowledge and are prone to fall into the sub-optimum. Besides, the number of editing operations is limited in the previous methods, which decreases the diversity of the augmented data and thus restricts the performance gain. To overcome the above limitations, we p…
VisPlay: Self-Evolving Vision-Language Models from Images
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Washington University in St. Louis（华盛顿大学圣路易斯分校）；University of Maryland（马里兰大学）；National University of Singapore（新加坡国立大学）
Conflict-Aware Additive Guidance for Flow Models under Compositional Rewards
Smart Systems Institute, National University of Singapore, Singapore（新加坡国立大学智能系统研究所）；Faculty of Computing, Harbin Institute of Technology, Harbin, China（哈尔滨工业大学计算机学院）；School of Computing, National University of Singapore, Singapore（新加坡国立大学计算机学院）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
We consider the problem of combining machine learning models to perform higher-level cognitive tasks with clear specifications. We propose the novel problem of Visual Discrimination Puzzles (VDP) that requires finding interpretable discriminators that classify images according to a logical specification. Humans can solve these puzzles with ease and they give robust, verifiable, and interpretable discriminators as answers. We propose a compositional neurosymbolic framework that combines a neural network to detect objects and relationships with a symbolic learner that finds interpretable discrim…
