Multi-objective alignment aims to align LLM responses with multiple human preference objectives. Among existing methods, guiding the generation of frozen LLMs through autoregressive reward models (ARMs) to accomplish multi-objective test-time alignment is a low-cost solution. However, these methods typically rely on independent parameters for each preference objective, either by training ARMs independently across preference dimensions, which neglects interactions among preference features, or by training a single ARM with separate feature extraction modules for each preference, which can cause feature entanglement. Both strategies can result in misalignment between generated outputs and user preferences. To address this limitation, we propose Preference-Modulated \& Shared Low-Rank Adaptation (MoSLoRA) for ARM training, which first extracts shared features via a preference-agnostic module and then applies affine transformations to shared features via a preference modulation module conditioned on mixed preference vectors. This design mitigates feature entanglement and enables precise control over preference trade-offs during inference. Building on this, we introduce the Unified Autoregressive Reward Model (UniARM), a novel framework for multi-objective test-time alignment. UniARM jointly models all preference dimensions in a single parameter space, eliminating the need for independent parameters for each preference objective. es on larger-scale LLMs, enhancing its practical usability.
Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editing
SKLCCSE, Beihang University（北航智能计算与复杂系统研究院）；National University of Singapore（新加坡国立大学）；Zhongguancun Laboratory, Beijing（中关村实验室）；Institute of Dataspace, Hefei（合肥数据空间研究院）；Nanyang Technological University（南洋理工大学）
MRO: Enhancing Reasoning in Diffusion Language Models via Multi-Reward Optimization
Northeastern University（东北大学）；ByteDance（字节跳动）；NiuTrans Research（小牛翻译研究院）；CAS Key Laboratory of Behavioral Science, Institute of Psychology, CAS（中国科学院行为科学重点实验室，中国科学院心理研究所）；Kunming University of Science and Technology（昆明理工大学）
Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step
The Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）；Shanghai AI Laboratory（上海人工智能实验室）；University of Electronic Science and Technology of China（电子科技大学）
TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院（KAIST））；Kakao Brain；Korea University（高丽大学）；University of Illinois at Urbana-Champaign (UIUC)（伊利诺伊大学厄巴纳-香槟分校（UIUC））
Are We Ready for RL in Text-to-3D Generation? A Progressive Investigation
Northwestern Polytechnical University（西北工业大学）；Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Lab（上海人工智能实验室）；The Hong Kong University of Science and Technology（香港科技大学）
