语言大模型 / LLM

Base Models Beat Aligned Models at Randomness and Creativity

Peter West, Christopher Potts

机构 * Stanford University（斯坦福大学） ； University of British Columbia（不列颠哥伦比亚大学） 纠错

专题命中 后训练与偏好优化 ：LLM(abstract)；language model(abstract)；分类 cs.CL

The Majority is not always right: RL training for solution aggregation

Wenting Zhao, Pranjal Aggarwal, Swarnadeep Saha, Asli Celikyilmaz, Jason Weston, Ilia Kulikov

机构 * FAIR at Meta（Meta旗下的FAIR） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL

Counterfactual Reward Model Training for Bias Mitigation in Multimodal Reinforcement Learning

Sheryl Mathew, N Harshit

专题命中 后训练与偏好优化 ：language model(abstract)；RLHF(abstract)；分类 cs.LG

Coarse-to-Fine Process Reward Modeling for Mathematical Reasoning

Yulan Hu, Sheng Ouyang, Jinman Zhao, Yong Liu

机构 * Renmin University of China（中国人民大学） ； Gaoling School of Artificial Intelligence（甘肃学校人工智能学院） ； University of Toronto（多伦多大学） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.AI

Discrete Optimization of Min-Max Violation and its Applications Across Computational Sciences

Cheikh Ahmed, Mahdi Mostajabdaveh, Samin Aref, Zirui Zhou

专题命中 后训练与偏好优化 ：language model(abstract)；post-training(abstract)；分类 cs.AI

G$^2$RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance

Yongxin Guo, Wenbo Deng, Zhenglin Cheng, Xiaoying Tang

机构 * School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong（科学与工程学院，香港中文大学（深圳）） ； Alibaba Group（阿里巴巴集团） ； School of Engineering, Westlake University（工程学院，西湖大学） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.AI

Wisdom of the Crowd: Reinforcement Learning from Coevolutionary Collective Feedback

Wenzhen Yuan, Shengji Tang, Weihao Lin, Jiacheng Ruan, Ganqu Cui, Bo Zhang, Tao Chen, Ting Liu, Yuzhuo Fu, Peng Ye, Lei Bai

机构 * Shanghai AI Lab（上海AI实验室） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.AI

EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity

Xingjian Zhang, Siwei Wen, Wenjun Wu, Lei Huang

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.AI

Libra: Assessing and Improving Reward Model by Learning to Think

Meng Zhou, Bei Li, Jiahao Liu, Xiaowen Shi, Yang Bai, Rongxiang Weng, Jingang Wang, Xunliang Cai

机构 * Meituan（美团） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL

Comments Work In Progress

GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment

Yuancheng Xu, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, Sumitra Ganesh

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL

Comments Published at the Thirteenth International Conference on Learning Representations (ICLR 2025)

Data-Centric Human Preference with Rationales for Direct Preference Alignment

Hoang Anh Just, Ming Jin, Anit Sahu, Huy Phan, Ruoxi Jia

机构 * Virginia Tech（弗吉尼亚理工大学） ； Amazon（亚马逊公司） 纠错

专题命中 后训练与偏好优化 ：language model(abstract)；preference optimization(abstract)；分类 cs.LG

Comments Data-Centric Human Preference with Rationales for Direct Preference Alignment

Test-Time Reasoning Through Visual Human Preferences with VLMs and Soft Rewards

Alexander Gambashidze, Konstantin Sobolev, Andrey Kuznetsov, Ivan Oseledets

专题命中 后训练与偏好优化 ：language model(abstract)；preference optimization(abstract)；分类 cs.AI

Comments We are withdrawing this paper because the main contributions and methodology have significantly changed after further research and experimental updates. The current version no longer reflects our results and main contribution / topic

Reflective Verbal Reward Design for Pluralistic Alignment

Carter Blair, Kate Larson, Edith Law

机构 * University of Waterloo（滑铁卢大学） 纠错

专题命中 后训练与偏好优化 ：language model(abstract)；RLHF(abstract)；分类 cs.AI

Comments 9 pages, 3 figures, accepted to the IJCAI 2025 Human-Centred AI track. Project repository at: https://osf.io/8yxf2/

Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminator

Kaiwen Zheng, Yongxin Chen, Huayu Chen, Guande He, Ming-Yu Liu, Jun Zhu, Qinsheng Zhang

机构 * Tsinghua University（清华大学） ； The University of Texas at Austin（得克萨斯大学） ； NVIDIA（英伟达） 纠错

专题命中 后训练与偏好优化 ：pretraining(abstract)；preference optimization(abstract)；分类 cs.LG

Comments ICML 2025 Spotlight Project Page: https://research.nvidia.com/labs/dir/ddo/ Code: https://github.com/NVlabs/DDO

Think&Cite: Improving Attributed Text Generation with Self-Guided Tree Search and Progress Reward Modeling

Junyi Li, Hwee Tou Ng

机构 * Department of Computer Science, National University of Singapore（计算机科学系，新加坡国立大学） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL

Comments ACL 2025

Dynamic Epistemic Friction in Dialogue

Timothy Obiso, Kenneth Lai, Abhijnan Nath, Nikhil Krishnaswamy, James Pustejovsky

机构 * Brandeis University（布兰迪斯大学） ； Colorado State University（科罗拉多州立大学） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL

Comments 11 pages, 2 figures, 2 tables, CoNLL 2025

AI as Decision-Maker: Ethics and Risk Preferences of LLMs

Shumiao Ouyang, Hayong Yun, Xingjian Zheng

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.AI

FreePRM: Training Process Reward Models Without Ground Truth Process Labels

Lin Sun, Chuang Liu, Xiaofeng Ma, Tao Yang, Weijia Lu, Ning Wu

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL

From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling

Zhengyu Chen, Yudong Wang, Teng Xiao, Ruochen Zhou, Xuesheng Yang, Wei Wang, Zhifang Sui, Jingang Wang

机构 * Meituan Inc.（美团公司） ； National Key Laboratory for Multimedia Information Processing, Peking University（北京大学多媒体信息处理国家重点实验室） ； Pennsylvania State University（宾夕法尼亚大学） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL

InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model

Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, Kai Chen, Dahua Lin, Jiaqi Wang

机构 * Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） ； The Chinese University of Hong Kong（香港中文大学） ； Shanghai Jiao Tong University（上海交通大学） ； Nanjing University（南京大学） ； Fudan University（复旦大学） ； Nanyang Technological University（南洋理工大学） 纠错

专题命中 后训练与偏好优化 ：language model(abstract)；instruction tuning(abstract)；分类 cs.CL

Comments ACL 2025 Findings

WorldPM: Scaling Human Preference Modeling

Binghai Wang, Runji Lin, Keming Lu, Le Yu, Zhenru Zhang, Fei Huang, Chujie Zheng, Kai Dang, Yang Fan, Xingzhang Ren, An Yang, Binyuan Hui, Dayiheng Liu, Tao Gui, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang, Bowen Yu, Jingren Zhou, Junyang Lin

专题命中 后训练与偏好优化 ：language model(abstract)；RLHF(abstract)；分类 cs.CL

Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models

Zae Myung Kim, Chanwoo Park, Vipul Raheja, Suin Kim, Dongyeop Kang

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL

Comments Code and data: https://github.com/minnesotanlp/mpo

Explaining Relationships Among Research Papers

Xiangci Li, Jessica Ouyang

机构 * Department of Computer Science University of Texas at Dallas（计算机科学系 德克萨斯大学达拉斯分校） 纠错

专题命中 后训练与偏好优化 ：LLM(abstract)；prompting(abstract)；分类 cs.CL

Journal ref https://aclanthology.org/2025.coling-main.73/

S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning

Ni Mu, Yao Luan, Yiqin Yang, Bo Xu, Qing-shan Jia

机构 * Department of Automation, Tsinghua University（清华大学自动化系） ； Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所） 纠错

专题命中 后训练与偏好优化 ：pretraining(abstract)；preference optimization(abstract)；分类 cs.AI

Comments IJCAI 2025

DMRL: Data- and Model-aware Reward Learning for Data Extraction

Zhiqiang Wang, Ruoxi Cheng

机构 * Beijing Electronic Science and Technology Institute（北京电子科学技术研究所） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.LG

Comments Data- and Model-aware Reward Learning for Data Extraction. arXiv admin note: substantial text overlap with arXiv:2503.18991

R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning

Yi-Fan Zhang, Xingyu Lu, Xiao Hu, Chaoyou Fu, Bin Wen, Tianke Zhang, Changyi Liu, Kaiyu Jiang, Kaibing Chen, Kaiyu Tang, Haojie Ding, Jiankang Chen, Fan Yang, Zhang Zhang, Tingting Gao, Liang Wang

机构 * CASIA（中国科学院自动化研究所） ； THU（清华大学） ； KuaiShou（快手） ； NJU（南京大学） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL

Comments Home page: https://github.com/yfzhang114/r1_reward

Improving Generalization in Intent Detection: GRPO with Reward-Based Curriculum Sampling

Zihao Feng, Xiaoxue Wang, Ziwei Bai, Donghang Su, Bowen Wu, Qun Yu, Baoxun Wang

专题命中 后训练与偏好优化 ：LLM(abstract)；SFT(abstract)；分类 cs.CL

Adversarial Training of Reward Models

Alexander Bukharin, Haifeng Qian, Shengyang Sun, Adithya Renduchintala, Soumye Singhal, Zhilin Wang, Oleksii Kuchaiev, Olivier Delalleau, Tuo Zhao

专题命中 后训练与偏好优化 ：language model(abstract)；RLHF(abstract)；分类 cs.LG

Comments 16 pages, 7 figures

PrefRAG: Preference-Driven Multi-Source Retrieval Augmented Generation

Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Jie Tang

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL

Comments 33 pages, 5 figures, 28 tables

DeepNote: Note-Centric Deep Retrieval-Augmented Generation

Ruobing Wang, Qingfei Zhao, Yukun Yan, Daren Zha, Yuxuan Chen, Shi Yu, Zhenghao Liu, Yixuan Wang, Shuo Wang, Xu Han, Zhiyuan Liu, Maosong Sun

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL

Comments 28 pages, 6 figures, 21 tables