大模型对齐与安全

RSPO: Regularized Self-Play Alignment of Large Language Models

Xiaohang Tang, Sangwoong Yoon, Seongho Son, Huizhuo Yuan, Quanquan Gu, Ilija Bogunovic

机构 * University College London（伦敦大学学院） ； University of California, Los Angeles（加州大学洛杉矶分校） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

Comments Preprint

Accelerating RLHF Training with Reward Variance Increase

Zonglin Yang, Zhexuan Gu, Houduo Qi, Yancheng Yuan

机构 * The Hong Kong Polytechnic University（香港理工大学） 纠错

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.AI、cs.LG

Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment

Xiaotian Zhang, Ruizhe Chen, Yang Feng, Zuozhu Liu

机构 * Zhejiang University（浙江大学） ； Angelalign Technology Inc.（Angelalign技术公司） ； Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江省医学影像人工智能重点实验室） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments ACL Finding

Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding

Feifan Song, Shaohang Wei, Wen Luo, Yuxuan Fan, Tianyu Liu, Guoyin Wang, Houfeng Wang

机构 * State Key Laboratory of Multimedia Information Processing, School of Computer Science（多媒体信息处理国家重点实验室、计算机科学学院） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL 2025 Findings

SudoLM: Learning Access Control of Parametric Knowledge with Authorization Alignment

Qin Liu, Fei Wang, Chaowei Xiao, Muhao Chen

机构 * UC Davis（加州大学戴维斯分校） ； USC（南加州大学） ； UW-Madison（威斯康星大学麦迪逊分校） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments ACL 2025

Unlocking Recursive Thinking of LLMs: Alignment via Refinement

Haoke Zhang, Xiaobo Liang, Cunxiang Wang, Juntao Li, Min Zhang

机构 * Soochow University（苏州大学） ； Zhipu AI（智谱AI） ； Tsinghua University（清华大学） ； Key Laboratory of Data Intelligence and Advanced Computing, Soochow University（苏州大学数据智能与先进计算重点实验室） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted to the Findings of ACL 2025

Multi-objective Large Language Model Alignment with Hierarchical Experts

Zhuo Li, Guodong Du, Weiyang Guo, Yigeng Zhou, Xiucheng Li, Wenya Wang, Fangming Liu, Yequan Wang, Deheng Ye, Min Zhang, Jing Li

机构 * Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳）） ； Nanyang Technological University（南洋理工大学） ； Peng Cheng Laboratory（鹏城实验室） ； Beijing Academy of Artificial Intelligence（北京人工智能研究院） ； Tencent（腾讯） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment

Jia-Nan Li, Jian Guan, Songhao Wu, Wei Wu, Rui Yan

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment

Wenqiao Zhu, Ji Liu, Lulu Wang, Jun Wu, Yulun Zhang

专题命中 偏好对齐 ：alignment(title)；DPO(abstract)；分类 cs.AI、cs.LG

Comments 18 pages, to appear in ACL'25

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model

Baijiong Lin, Weisen Jiang, Yuancheng Xu, Hao Chen, Ying-Cong Chen

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

Comments Accepted by ICML 2025

Optimizing RLHF Training for Large Language Models with Stage Fusion

Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, Xin Jin

机构 * School of Computer Science, Peking University（北京大学计算机学院） 纠错

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.LG

VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models

Dahun Kim, AJ Piergiovanni, Ganesh Mallya, Anelia Angelova

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments CVPR 2025, project page at https://github.com/google-deepmind/video_comp

Preference-Based Alignment of Discrete Diffusion Models

Umberto Borso, Davide Paglieri, Jude Wells, Tim Rocktäschel

专题命中 偏好对齐 ：alignment(title)；DPO(abstract)；分类 cs.AI、cs.LG

Towards Federated RLHF with Aggregated Client Preference for LLMs

Feijie Wu, Xiaoze Liu, Haoyu Wang, Xingchen Wang, Lu Su, Jing Gao

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.LG

Comments ICLR'25

Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment

Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Akshay Krishnamurthy, Dylan J. Foster

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

A Shared Low-Rank Adaptation Approach to Personalized RLHF

Renpu Liu, Peng Wang, Donghao Li, Cong Shen, Jing Yang

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.AI、cs.LG

Comments Published as a conference paper at AISTATS 2025

Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization

Juntao Dai, Taiye Chen, Yaodong Yang, Qian Zheng, Gang Pan

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.AI、cs.LG

Comments Published as a conference paper at ICLR 2025

Understanding the Effects of RLHF on the Quality and Detectability of LLM-Generated Texts

Beining Xu, Arkaitz Zubiaga

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.AI

Comments 14 pages, 3 figures

InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization

Yunan Wang, Jijie Li, Bo-Wen Zhang, Liangdong Wang, Guang Liu

专题命中 偏好对齐 ：DPO(title,abstract)；分类 cs.CL、cs.LG

PAD: Personalized Alignment of LLMs at Decoding-Time

Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, Zuozhu Liu

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments ICLR 2025

Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits

Tuhin Chakrabarty, Philippe Laban, Chien-Sheng Wu

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.CY

Comments ACM CHI 2025

Training-free Diffusion Model Alignment with Sampling Demons

Po-Hung Yeh, Kuang-Huei Lee, Jun-Cheng Chen

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

Comments 35 pages

Journal ref Proceedings of the Thirteenth International Conference on Learning Representations (ICLR 2025)

Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF

Nuoya Xiong, Aarti Singh

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.AI、cs.LG

On-the-fly Preference Alignment via Principle-Guided Decoding

Mingye Zhu, Yi Liu, Lei Zhang, Junbo Guo, Zhendong Mao

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted to ICLR 2025

SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters

Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, Zhaochun Ren, Vasant G Honavar

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.LG

Comments ICLR 2025

Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF

Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, Bo Dai

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.AI、cs.LG

Comments ICLR 2025

Efficient Alignment of Large Language Models via Data Sampling

Amrit Khera, Rajat Ghosh, Debojyoti Dutta

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.LG

Comments Original work accepted at NeurIPS Efficient Natural Language and Speech Processing Workshop. PMLR, 2024. Experiments with a larger model from a different family, Llama-30B have been added to the appendix for generalizability

e-SimFT: Alignment of Generative Models with Simulation Feedback for Pareto-Front Design Exploration

Hyunmin Cheong, Mohammadmehdi Ataei, Amir Hosein Khasahmadi, Pradeep Kumar Jayaraman

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model

Yueqin Yin, Shentao Yang, Yujia Xie, Ziyi Yang, Yuting Sun, Hany Awadalla, Weizhu Chen, Mingyuan Zhou

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.AI

Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment

Jianfei Zhang, Jun Bai, Bei Li, Yanmeng Wang, Rumei Li, Chenghua Lin, Wenge Rong

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Coling 2025