语言大模型 / LLM

Intra-Trajectory Consistency for Reward Modeling

Chaoyang Zhou, Shunyu Liu, Zengmao Wang, Di Wang, Rong-Cheng Tu, Bo Du, Dacheng Tao

机构 * Wuhan University（武汉大学） ； Nanyang Technological University（南洋理工大学） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments Under review

PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference

Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Juntao Dai, Boren Zheng, Tianyi Qiu, Jiayi Zhou, Kaile Wang, Boxuan Li, Sirui Han, Yike Guo, Yaodong Yang

机构 * Peking University（北京大学） ； The Hong Kong University of Science and Technology（香港科学与技术大学） ； Infinigence-AI 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL2025 Main, a sibling project to SafeRLHF and BeaverTails

Debiasing Online Preference Learning via Preference Feature Preservation

Dongyoung Kim, Jinsung Yoon, Jinwoo Shin, Jaehyung Kim

机构 * KAIST（韩国科学技术院） ； Google Cloud AI Research（谷歌云人工智能研究） ； Yonsei University（延世大学） 纠错

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments 20 page, 20 figures

OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation

Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Qiguang Chen, Zeyu Zhang, Yifeng Wang, Qianshuo Ye, Bernard Ghanem, Ping Luo, Guohao Li

机构 * The University of Hong Kong（香港大学） ； The University of Chicago（芝加哥大学） ； Northwestern University（西北大学） ； University of Cambridge（剑桥大学） 纠错

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Project Page: https://github.com/camel-ai/owl

Mutual-Taught for Co-adapting Policy and Reward Models

Tianyuan Shi, Canbin Huang, Fanqi Wan, Longguang Zhong, Ziyi Yang, Weizhou Shen, Xiaojun Quan, Ming Yan

机构 * School of Computer Science and Engineering, Sun Yat-sen University, China（计算机科学与工程学院，中山大学） ； Alibaba Group（阿里巴巴集团） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.AI、cs.LG

Comments Accepted to ACL 2025 (Main Conference)

Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding

Feifan Song, Shaohang Wei, Wen Luo, Yuxuan Fan, Tianyu Liu, Guoyin Wang, Houfeng Wang

机构 * State Key Laboratory of Multimedia Information Processing, School of Computer Science（多媒体信息处理国家重点实验室、计算机科学学院） 纠错

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL 2025 Findings

SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs

Michael J Ryan, Omar Shaikh, Aditri Bhagirath, Daniel Frees, William Held, Diyi Yang

机构 * Stanford University（斯坦福大学） ； Georgia Institute of Technology（佐治亚理工学院） 纠错

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments ACL 2025 Main Conference

Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment

Wen Yang, Junhong Wu, Chen Wang, Chengqing Zong, Jiajun Zhang

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI

Comments Camera ready version for ACL 2025 Findings

Self-Evolved Reward Learning for LLMs

Chenghua Huang, Zhizhen Fan, Lu Wang, Fangkai Yang, Pu Zhao, Zeqi Lin, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, Qi Zhang

机构 * School of Computer Science, Fudan University（复旦大学计算机学院） ； School of Computer Science, Peking University（北京大学计算机学院） ； Microsoft（微软公司） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments 23 pages,6 figures,Accepted to ICLR 2025

RLAE: Reinforcement Learning-Assisted Ensemble for LLMs

Yuqian Fu, Yuanheng Zhu, Jiajun Chai, Guojun Yin, Wei Lin, Qichao Zhang, Dongbin Zhao

机构 * Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所） ； School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院） 纠错

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

On Symmetric Losses for Robust Policy Optimization with Noisy Preferences

Soichiro Nishimori, Yu-Jie Zhang, Thanawat Lodkaew, Masashi Sugiyama

机构 * The University of Tokyo, Japan（东京大学） ； RIKEN AIP, Japan（日本理化学研究所AIP） 纠错

专题命中 后训练与偏好优化 ：language model(abstract)；RLHF(abstract)；preference optimization(abstract)；分类 cs.AI、cs.LG

TimeHC-RL: Temporal-aware Hierarchical Cognitive Reinforcement Learning for Enhancing LLMs' Social Intelligence

Guiyang Hou, Xing Gao, Yuchuan Wu, Xiang Huang, Wenqi Zhang, Zhe Zheng, Yongliang Shen, Jialu Du, Fei Huang, Yongbin Li, Weiming Lu

机构 * Zhejiang University（浙江大学） ； Tongyi Lab, Alibaba Group（阿里云实验室） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；post-training(abstract)；分类 cs.CL、cs.AI

Comments 22 pages, 12 figures

Beyond path selection: Better LLMs for Scientific Information Extraction with MimicSFT and Relevance and Rule-induced(R$^2$)GRPO

Ran Li, Shimin Di, Yuchen Liu, Chen Jing, Yu Qiu, Lei Chen

机构 * HKUST Hong Kong SAR, China（香港科技大学） ； SEU Jiangsu, China（江苏大学） ； HKUST(GZ) Guangzhou, China（香港科技大学（广州）） ； Zhipu AI Beijing, China（智谱AI） ； HKUST(GZ), HKUST Guangzhou, China（香港科技大学（广州）、香港科技大学） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；SFT(abstract)；分类 cs.CL、cs.AI

On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization

Wenlong Deng, Yi Ren, Muchen Li, Danica J. Sutherland, Xiaoxiao Li, Christos Thrampoulidis

机构 * University of British Columbia（不列颠哥伦比亚大学） ； Vector Institute（向量研究所） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.CL、cs.LG

From Lists to Emojis: How Format Bias Affects Model Alignment

Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, Tong Zhang

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments Working in progress

Disentangling Length Bias In Preference Learning Via Response-Conditioned Modeling

Jianfeng Cai, Jinhua Zhu, Ruopei Sun, Yue Wang, Li Li, Wengang Zhou, Houqiang Li

机构 * University of Science and Technology of China（中国科学技术大学） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Reinforced Interactive Continual Learning via Real-time Noisy Human Feedback

Yutao Yang, Jie Zhou, Junsong Li, Qianjun Pan, Bihao Zhan, Qin Chen, Xipeng Qiu, Liang He

机构 * School of Computer Science and Technology, East China Normal University（东华师范大学计算机科学与技术学院） ； Shanghai Innovation Institute（上海创新研究院） ； School of Computer Science, Fudan University（复旦大学计算机科学学院） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.AI、cs.LG

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model

Baijiong Lin, Weisen Jiang, Yuancheng Xu, Hao Chen, Ying-Cong Chen

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments Accepted by ICML 2025

A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future

Jialun Zhong, Wei Shen, Yanzeng Li, Songyang Gao, Hua Lu, Yicheng Chen, Yang Zhang, Wei Zhou, Jinjie Gu, Lei Zou

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Models

Yongjin Yang, Sihyeon Kim, Hojung Jung, Sangmin Bae, SangMook Kim, Se-Young Yun, Kimin Lee

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.AI、cs.LG

Comments ICLR 2025; Project Page available at : https://sprain02.github.io/FiFA/

PluralLLM: Pluralistic Alignment in LLMs via Federated Learning

Mahmoud Srewa, Tianyu Zhao, Salma Elmalaki

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Mitigating Preference Hacking in Policy Optimization with Pessimism

Dhawal Gupta, Adam Fisch, Christoph Dann, Alekh Agarwal

专题命中 后训练与偏好优化 ：language model(abstract)；RLHF(abstract)；preference optimization(abstract)；分类 cs.AI、cs.LG

Knowledge-Decoupled Synergetic Learning: An MLLM based Collaborative Approach to Few-shot Multimodal Dialogue Intention Recognition

Bin Chen, Yu Zhang, Hongfei Ye, Ziyi Huang, Hongyang Chen

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；post-training(abstract)；分类 cs.CL、cs.AI

SeqAR: Jailbreak LLMs with Sequential Auto-Generated Characters

Yan Yang, Zeguan Xiao, Xin Lu, Hongru Wang, Xuetao Wei, Hailiang Huang, Guanhua Chen, Yun Chen

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted by NAACL 2025

Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization

Wenkai Yang, Shiqi Shen, Guangyao Shen, Wei Yao, Yong Liu, Zhi Gong, Yankai Lin, Ji-Rong Wen

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI

Comments Accepted at ICLR 2025, camera-ready version

Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems

Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Bin Xu, Lei Hou, Juanzi Li

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 16 pages, 5 figures

Interpreting Language Reward Models via Contrastive Explanations

Junqi Jiang, Tom Bewley, Saumitra Mishra, Freddy Lecue, Manuela Veloso

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments Accepted at ICLR 2025 conference

Putting People in LLMs' Shoes: Generating Better Answers via Question Rewriter

Junhao Chen, Bowen Wang, Zhouqiang Jiang, Yuta Nakashima

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI

Comments 7 pages, 4 figures, 5 tables and accepted at AAAI 2025 Main Conference

Towards User-level Private Reinforcement Learning with Human Feedback

Jiaming Zhang, Mingxi Lei, Meng Ding, Mengdi Li, Zihang Xiang, Difei Xu, Jinhui Xu, Di Wang

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Rejected Dialects: Biases Against African American Language in Reward Models

Joel Mire, Zubin Trivadi Aysola, Daniel Chechelnitsky, Nicholas Deas, Chrysoula Zerva, Maarten Sap

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted to NAACL Findings 2025