大模型对齐与安全

Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model

Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, Yu Zhou, Deshan Sun, Deyu Zhou, Jian Zhou, Kaijun Tan, Kang An, Mei Chen, Wei Ji, Qiling Wu, Wen Sun, Xin Han, Yanan Wei, Zheng Ge, Aojie Li, Bin Wang, Bizhu Huang, Bo Wang, Brian Li, Changxing Miao, Chen Xu, Chenfei Wu, Chenguang Yu, Dapeng Shi, Dingyuan Hu, Enle Liu, Gang Yu, Ge Yang, Guanzhe Huang, Gulin Yan, Haiyang Feng, Hao Nie, Haonan Jia, Hanpeng Hu, Hanqi Chen, Haolong Yan, Heng Wang, Hongcheng Guo, Huilin Xiong, Huixin Xiong, Jiahao Gong, Jianchang Wu, Jiaoren Wu, Jie Wu, Jie Yang, Jiashuai Liu, Jiashuo Li, Jingyang Zhang, Junjing Guo, Junzhe Lin, Kaixiang Li, Lei Liu, Lei Xia, Liang Zhao, Liguo Tan, Liwen Huang, Liying Shi, Ming Li, Mingliang Li, Muhua Cheng, Na Wang, Qiaohui Chen, Qinglin He, Qiuyan Liang, Quan Sun, Ran Sun, Rui Wang, Shaoliang Pang, Shiliang Yang, Sitong Liu, Siqi Liu, Shuli Gao, Tiancheng Cao, Tianyu Wang, Weipeng Ming, Wenqing He, Xu Zhao, Xuelin Zhang, Xianfang Zeng, Xiaojia Liu, Xuan Yang, Yaqi Dai, Yanbo Yu, Yang Li, Yineng Deng, Yingming Wang, Yilei Wang, Yuanwei Lu, Yu Chen, Yu Luo, Yuchu Luo, Yuhe Yin, Yuheng Feng, Yuxiang Yang, Zecheng Tang, Zekai Zhang, Zidong Yang, Binxing Jiao, Jiansheng Chen, Jing Li, Shuchang Zhou, Xiangyu Zhang, Xinhao Zhang, Yibo Zhu, Heung-Yeung Shum, Daxin Jiang

机构 * StepFun 纠错

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Comments 36 pages, 14 figures

Generative Verifiers: Reward Modeling as Next-Token Prediction

Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, Rishabh Agarwal

机构 * Google DeepMind（谷歌DeepMind） ； University of Toronto（多伦多大学） ； Carnegie Mellon University（卡内基梅隆大学） 纠错

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.LG

Comments ICLR 2025

Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models

Michihiro Yasunaga, Luke Zettlemoyer, Marjan Ghazvininejad

机构 * FAIR at Meta（Meta FAIR实验室） 纠错

专题命中 偏好对齐 ：safety(abstract)；分类 cs.AI

Comments Dataset available at https://github.com/facebookresearch/multimodal_rewardbench

Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework

Yannick Metz, David Lindner, Raphaël Baur, Mennatallah El-Assady

机构 * University of Konstanz（康斯坦茨大学） ； ETH Zurich（苏黎世联邦理工学院） 纠错

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.LG

SPPD: Self-training with Process Preference Learning Using Dynamic Value Margin

Hao Yi, Qingyang Li, Yulan Hu, Fuzheng Zhang, Di Zhang, Yong Liu

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.AI

Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuning

Mohit Raghavendra, Junmo Kang, Alan Ritter

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.LG

Preference Optimization for Reasoning with Pseudo Feedback

Fangkai Jiao, Geyang Guo, Xingxing Zhang, Nancy F. Chen, Shafiq Joty, Furu Wei

机构 * Nanyang Technological University（南洋理工大学） ； Microsoft Research（微软研究院） ； Georgia Institute of Technology（佐治亚理工学院） ； Salesforce Research（Salesforce研究院） 纠错

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Comments 28 pages, 11 figures. ICLR 2025

On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validation

Can Cui, Zichong Yang, Yupeng Zhou, Juntong Peng, Sung-Yeon Park, Cong Zhang, Yunsheng Ma, Xu Cao, Wenqian Ye, Yiheng Feng, Jitesh Panchal, Lingxi Li, Yaobin Chen, Ziran Wang

机构 * Purdue University（普渡大学） ； University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） ； University of Virginia（弗吉尼亚大学） 纠错

专题命中 偏好对齐 ：safety(abstract)；分类 cs.AI

Comments arXiv admin note: substantial text overlap with arXiv:2410.15281

EmoAssist: Emotional Assistant for Visual Impairment Community

Xingyu Qi, He Li, Linjie Li, Zhenyu Wu

机构 * Beijing University of Posts and Telecommunications（北京邮电大学） ； AI2Robotics 纠错

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CY

Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models

Xin Zhou, Yiwen Guo, Ruotian Ma, Tao Gui, Qi Zhang, Xuanjing Huang

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.AI

Optimizing Knowledge Integration in Retrieval-Augmented Generation with Self-Selection

Yan Weng, Fengbin Zhu, Tong Ye, Haoyan Liu, Fuli Feng, Tat-Seng Chua

机构 * University of Science and Technology of China（中国科学技术大学） ； National University of Singapore（新加坡国立大学） ； Institute of Dataspace, Hefei Comprehensive National Science Center（合肥综合性国家科学中心数据空间研究院） 纠错

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Comments 12 pages, 6 figures

Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimator

Zhuotong Chen, Fang Liu, Xuan Zhu, Yanjun Qi, Mohammad Ghavamzadeh

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.AI

ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization

Yinjie Wang, Ling Yang, Guohao Li, Mengdi Wang, Bryon Aragam

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Comments Project: https://github.com/Gen-Verse/ScoreFlow

PILAF: Optimal Human Preference Sampling for Reward Modeling

Yunzhen Feng, Ariel Kwiatkowski, Kunhao Zheng, Julia Kempe, Yaqi Duan

机构 * NYU（纽约大学） ； Meta FAIR 纠错

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.LG

Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning

Udita Ghosh, Dripta S. Raychaudhuri, Jiachen Li, Konstantinos Karydis, Amit Roy-Chowdhury

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.LG

RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data

Chenglong Wang, Yang Gan, Yifu Huo, Yongyu Mu, Murun Yang, Qiaozhi He, Tong Xiao, Chunliang Zhang, Tongran Liu, Quan Du, Di Yang, Jingbo Zhu

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL

Comments Accepted by AAAI 2025

CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs

Jinlan Fu, Shenzhen Huangfu, Hao Fei, Xiaoyu Shen, Bryan Hooi, Xipeng Qiu, See-Kiong Ng

机构 * National University of Singapore（新加坡国立大学） ； Fudan University（复旦大学） ； Digital Twin Institute（数字孪生研究院） ； Eastern Institute of Technology（东方理工学院） 纠错

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Comments Accepted by ICLR 2025

Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives

Hao Sun, Yunyi Shen, Jean-Francois Ton

机构 * University of Cambridge（剑桥大学） ； Massachusetts Institute of Technology（麻省理工学院） ； ByteDance Research（字节跳动研究院） 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.AI

Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis

Qining Zhang, Honghao Wei, Lei Ying

机构 * University of Michigan（密歇根大学） ； Washington State University（华盛顿州立大学） 纠错

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.LG

DNA 1.0 Technical Report

Jungyup Lee, Jemin Kim, Sang Park, SeungJae Lee

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion

Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Bill Wu, Eugene Choi, Chris Cremer, Arash Ahmadian, Yash Chandak, Mohammad Gheshlaghi Azar, Olivier Pietquin, Matthieu Geist

机构 * Cohere（科here公司） 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.LG

Comments EMNLP 2024

SYNAPSE: SYmbolic Neural-Aided Preference Synthesis Engine

Sadanand Modak, Noah Patton, Isil Dillig, Joydeep Biswas

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.LG

Comments Accepted (oral) at AAAI 25

MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answering

Janak Kapuriya, Chhavi Kirtani, Apoorv Singh, Jay Saraf, Naman Lal, Jatin Kumar, Adarsh Raj Shivam, Astha Verma, Avinash Anand, Rajiv Ratn Shah

机构 * IIIT Delhi（德里印度信息技术学院） 纠错

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.AI

LoRA-LiteE: A Computationally Efficient Framework for Chatbot Preference-Tuning

Yahe Yang, Chunliang Tao, Xiaojing Fan

机构 * George Washington University（乔治·华盛顿大学） ； New York University（纽约大学） 纠错

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL

Distilling Desired Comments for Enhanced Code Review with Large Language Models

Yongda Yu, Lei Zhang, Guoping Rong, Haifeng Shen, Jiahao Zhang, Haoxiang Yan, Guohao Shi, Dong Shao, Ruiqi Pan, Yuan Li, Qiushi Wang, Zhao Tian

机构 * Software Institute Nanjing University（南京大学软件学院） ； Faculty of Science and Engineering Southern Cross University（南十字星大学科学与工程学院） ； Huawei Technologies Co., Ltd.（华为技术有限公司） 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.AI

Comments 12 pages, 9 figures

Qwen2.5 Technical Report

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL

Plug-and-Play Training Framework for Preference Optimization

Jingyuan Ma, Rui Li, Zheng Li, Lei Sha, Zhifang Sui

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Comments 12 pages, 9 figures

Online Learning from Strategic Human Feedback in LLM Fine-Tuning

Shugang Hao, Lingjie Duan

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.AI

Teaching LLMs to Refine with Tools

Dian Yu, Yuheng Zhang, Jiahao Xu, Tian Liang, Linfeng Song, Zhaopeng Tu, Haitao Mi, Dong Yu

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

The Superalignment of Superhuman Intelligence with Large Language Models

Minlie Huang, Yingkang Wang, Shiyao Cui, Pei Ke, Jie Tang

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL

Comments Under review of Science China