Diamond Maps: Efficient Reward Alignment via Stochastic Flow Maps
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Carnegie Mellon University（卡内基梅隆大学）
Peter Holderrieth, Douglas Chen, Luca Eyring, Ishin Shah, Giri Anantharaman, Yutong He, Zeynep Akata, Tommi Jaakkola, Nicholas Matthew Boffi, Max Simchowitz
Flow and diffusion models produce high-quality samples, but adapting them to user preferences or constraints post-training remains costly and brittle, a challenge commonly called reward alignment. We argue that efficient reward alignment should be a property of the generative model itself, not an afterthought, and redesign the model for adaptability. We propose "Diamond Maps", stochastic flow map models that enable efficient and accurate alignment to arbitrary rewards at inference time. Diamond Maps amortize many simulation steps into a single-step sampler, like flow maps, while preserving the stochasticity required for optimal reward alignment. This design makes search, Sequential Monte Carlo, and guidance scalable by enabling efficient and consistent estimation of the value function. Our experiments show that Diamond Maps can be learned efficiently via distillation from GLASS Flows, achieve stronger reward alignment performance, and scale better than existing methods. Our results point toward a practical route to generative models that can be rapidly adapted to arbitrary preferences and constraints at inference time.
Inference-Time Alignment of Diffusion Models with Direct Noise Optimization
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Hupan Lab, Zhejiang Province（浙江省湖畔实验室）；University of Minnesota, USA（美国明尼苏达大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
Harness Local Rewards for Global Benefits: Effective Text-to-Video Generation Alignment with Patch-level Reward Models
School of Computer Science, Carnegie Mellon University（卡内基梅隆大学计算机科学学院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Serendipity One Inc.（Serendipity One公司）
Alignment of Diffusion Model and Flow Matching for Text-to-Image Generation
Department of Statistics, University of California, Los Angeles（加州大学洛杉矶分校统计学系）；Department of Industrial and Systems Engineering, University of Minneasota（明尼苏达大学工业与系统工程系）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
One obstacle to applying reinforcement learning algorithms to real-world problems is the lack of suitable reward functions. Designing such reward functions is difficult in part because the user only has an implicit understanding of the task objective. This gives rise to the agent alignment problem: how do we create agents that behave in accordance with the user's intentions? We outline a high-level research direction to solve the agent alignment problem centered around reward modeling: learning a reward function from interaction with the user and optimizing the learned reward function with rei…
Stanford University（斯坦福大学）；University of Washington（华盛顿大学）
Modern multi-agent reinforcement learning frameworks rely on centralized training and reward shaping to perform well. However, centralized training and dense rewards are not readily available in the real world. Current multi-agent algorithms struggle to learn in the alternative setup of decentralized training or sparse rewards. To address these issues, we propose a self-supervised intrinsic reward ELIGN - expectation alignment - inspired by the self-organization principle in Zoology. Similar to how animals collaborate in a decentralized manner with those in their vicinity, agents trained with …
PREFINE: Preference-Based Implicit Reward and Cost Fine-Tuning for Safety Alignment
TCS Research, \ of CSE, IIT Madras India；Department of Computing Science, \ of Alberta Canada；Qatar Computing Research Institute, \ Bin Khalifa University Qatar；Department of Data Science \& AI, Wadhwani School of Data Science \& AI, IIT Madras India；TCS Research, \ of CSE, IIT Madras；Department of Computing Science, \ of Alberta；Qatar Computing Research Institute, \ Bin Khalifa University；Department of Data Science \& AI, Wadhwani School of Data Science \& AI, IIT Madras
The Reward Model Selection Crisis in Personalized Alignment
School of Informatics, University of Edinburgh（信息学院，爱丁堡大学）；Agency for Science, Technology and Research (A*STAR)（科技研究局）；Samsung AI Research Center（三星人工智能研究中心）
Does LLM Alignment Really Need Diversity? An Empirical Study of Adapting RLVR Methods for Moral Reasoning
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Microsoft Research（微软研究院）；University of Michigan（密歇根大学）；Shanghai Jiao Tong University（上海交通大学）；CUHKSZ（香港中文大学（深圳））；THU（清华大学）
