语言大模型 / LLM

A First-Order Logic-Based Alternative to Reward Models in RLHF

基于一阶逻辑的强化学习人类反馈中奖励模型的替代方案

Chunjin Jian, Xinhua Zhu

机构 * School of Computer Science（计算机科学系） ； Engineering Guangxi Normal University Guilin, China（广西师范大学） 纠错

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；SFT(abstract)

AI总结 本文提出基于一阶逻辑的奖励机制，替代传统奖励建模，通过逻辑一致性提升模型对齐性能，实验显示其在性能和鲁棒性上优于标准微调方法。

SAFENLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database Interfaces

Ruiheng Liu, XiaoBing Chen, Jinyu Zhang, Qiongwen Zhang, Yu Zhang, Bailong Yang

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Comments AAAI 2026 Extended Version

AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesis

Xuanzhong Chen, Zile Qiao, Guoxin Chen, Liangcai Su, Zhen Zhang, Xinyu Wang, Pengjun Xie, Fei Huang, Jingren Zhou, Yong Jiang

机构 * Tongyi Lab（通义实验室） ； Alibaba Group（阿里巴巴集团） 纠错

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；post-training(abstract)

Comments https://tongyi-agent.github.io/blog/introducing-tongyi-deep-research/

SynCast: Synergizing Contradictions in Precipitation Nowcasting via Diffusion Sequential Preference Optimization

Kaiyi Xu, Junchao Gong, Wenlong Zhang, Ben Fei, Lei Bai, Wanli Ouyang

机构 * University of Science and Technology of China（中国科学技术大学） ； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） ； Shanghai Jiao Tong University（上海交通大学） ； Chinese University of Hong Kong（香港中文大学） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；post-training(abstract)

Ask a Strong LLM Judge when Your Reward Model is Uncertain

Zhenghao Xu, Qin Lu, Qingru Zhang, Liang Qiu, Ilgee Hong, Changlong Yu, Wenlin Yao, Yao Liu, Haoming Jiang, Lihong Li, Hyokun Yun, Tuo Zhao

机构 * Georgia Institute of Technology（佐治亚理工学院） ； Amazon（亚马逊） 纠错

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments NeurIPS 2025, 18 pages

KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity

Gholamali Aminian, Amir R. Asadi, Idan Shenfeld, Youssef Mroueh

机构 * The Alan Turing Institute（艾伦·图灵研究所） ； Statistical Laboratory（统计实验室） ； University of Cambridge（剑桥大学） ； Massachusetts Institute of Technology（麻省理工学院） ； IBM Research USA（IBM美国研究） 纠错

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments Extra experiments are added in new version

Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences

Keertana Chidambaram, Karthik Vinay Seetharaman, Vasilis Syrgkanis

机构 * Stanford University（斯坦福大学） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences

Keertana Chidambaram, Karthik Vinary Seetharaman, Vasilis Syrgkanis

机构 * Stanford University（斯坦福大学） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning

Jared Joselowitz, Ritam Majumdar, Arjun Jagota, Matthieu Bou, Nyal Patel, Satyapriya Krishna, Sonali Parbhoo

机构 * Imperial College London（伦敦帝国学院） ； Harvard University（哈佛大学） 纠错

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments Published as a conference paper at COLM 2025

Mind the Gap: Linguistic Divergence and Adaptation Strategies in Human-LLM Assistant vs. Human-Human Interactions

Fulei Zhang, Zhou Yu

机构 * Amazon.com Inc.（亚马逊公司） 纠错

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；post-training(abstract)

Comments Accepted to The Second Workshop on Generative AI for E-commerce (GenAIECommerce '25), held September 22, 2025, in Prague, Czech Republic

StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Models

Chenyu Zhou, Tianyi Xu, Jianghao Lin, Dongdong Ge

机构 * Shanghai Jiao Tong University（上海交通大学） 纠错

专题命中 后训练与偏好优化 ：language model(title,abstract)；LLM(abstract)；large language model(abstract)；preference optimization(abstract)

Direct Judgement Preference Optimization

Peifeng Wang, Austin Xu, Yilun Zhou, Caiming Xiong, Shafiq Joty

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments EMNLP 2025

Balanced Actor Initialization: Stable RLHF Training of Distillation-Based Reasoning Models

Chen Zheng, Yiyuan Ma, Yuan Yang, Deyi Liu, Jing Liu, Zuquan Song, Yuxin Song, Cheng Ren, Hang Zhu, Xin Liu, Yiyuan Ma, Siyuan Qiao, Xun Zhou, Liang Xiang, Yonghui Wu

机构 * Peking University（北京大学） 纠错

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；instruction tuning(abstract)

On Negative-aware Preference Optimization for Recommendation

Chenlu Ding, Daoxuan Liu, Jiancan Wu, Xingyu Hu, Junkang Wu, Haitao Wang, Yongkang Wang, Xingxing Wang, Xiang Wang

机构 * USTC（中国科学技术大学） ； Meituan（美团） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

RLHF Fine-Tuning of LLMs for Alignment with Implicit User Feedback in Conversational Recommenders

Zhongheng Yang, Aijia Sun, Yushang Zhao, Yinuo Yang, Dannier Li, Chengrui Zhou

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Attack the Messages, Not the Agents: A Multi-round Adaptive Stealthy Tampering Framework for LLM-MAS

Bingyu Yan, Ziyi Zhou, Xiaoming Zhang, Chaozhuo Li, Ruilin Zeng, Yirui Qi, Tianbo Wang, Litian Zhang

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decomposition

Danqing Shi, Furui Cheng, Tino Weinkauf, Antti Oulasvirta, Mennatallah El-Assady

机构 * Aalto University（阿alto大学） ； ETH Zürich（苏黎世联邦理工学院） ； KTH Royal Institute of Technology（皇家理工学院） 纠错

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Harnessing RLHF for Robust Unanswerability Recognition and Trustworthy Response Generation in LLMs

Shuyuan Lin, Lei Duan, Philip Hughes, Yuxuan Sheng

机构 * Sichuan University of Science and Engineering（四川理工学院） ； Zagazig University（扎加齐亚大学） 纠错

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback

Kyuyoung Kim, Ah Jeong Seo, Hao Liu, Jinwoo Shin, Kimin Lee

机构 * KAIST（韩国科学技术院） ； UC Berkeley（加州大学伯克利分校） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments EMNLP 2024 Findings

Expectation Confirmation Preference Optimization for Multi-Turn Conversational Recommendation Agent

Xueyang Feng, Jingsen Zhang, Jiakai Tang, Wei Li, Guohao Cai, Xu Chen, Quanyu Dai, Yue Zhu, Zhenhua Dong

机构 * Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China（中国人民大学北京校区人工智能学院） ； Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大型模型与智能治理研究重点实验室） ； Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（下一代智能搜索与推荐工程研究中心，教育部） ； Huawei Noah’s Ark Lab, China（华为诺亚方舟实验室） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments Accepted to Findings of ACL 2025

Mastering Da Vinci Code: A Comparative Study of Transformer, LLM, and PPO-based Agents

LeCheng Zhang, Yuanshi Wang, Haotian Shen, Xujie Wang

机构 * Westlake College, Westlake University（西lake学院，西lake大学） 纠错

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model

Ailin Huang, Bingxin Li, Bruce Wang, Boyong Wu, Chao Yan, Chengli Feng, Heng Wang, Hongyu Zhou, Hongyuan Wang, Jingbei Li, Jianjian Sun, Joanna Wang, Mingrui Chen, Peng Liu, Ruihang Miao, Shilei Jiang, Tian Fei, Wang You, Xi Chen, Xuerui Yang, Yechang Huang, Yuxiang Zhang, Zheng Ge, Zheng Gong, Zhewei Huang, Zixin Zhang, Bin Wang, Bo Li, Buyun Ma, Changxin Miao, Changyi Wan, Chen Xu, Dapeng Shi, Dingyuan Hu, Enle Liu, Guanzhe Huang, Gulin Yan, Hanpeng Hu, Haonan Jia, Jiahao Gong, Jiaoren Wu, Jie Wu, Jie Yang, Junzhe Lin, Kaixiang Li, Lei Xia, Longlong Gu, Ming Li, Nie Hao, Ranchen Ming, Shaoliang Pang, Siqi Liu, Song Yuan, Tiancheng Cao, Wen Li, Wenqing He, Xu Zhao, Xuelin Zhang, Yanbo Yu, Yinmin Zhong, Yu Zhou, Yuanwei Liang, Yuanwei Lu, Yuxiang Yang, Zidong Yang, Zili Zhang, Binxing Jiao, Heung-Yeung Shum, Jiansheng Chen, Jing Li, Xiangyu Zhang, Xinhao Zhang, Yibo Zhu, Daxin Jiang, Shuchang Zhou, Chen Hu

机构 * Step-Audio Team（Step-Audio团队） 纠错

专题命中 后训练与偏好优化 ：language model(title,abstract)；LLM(abstract)；post-training(abstract)；preference optimization(abstract)

Comments 12 pages, 3 figures

Mitigating Spurious Correlations in LLMs via Causality-Aware Post-Training

Shurui Gui, Shuiwang Ji

专题命中 后训练与偏好优化 ：post-training(title,abstract)；large language model(abstract)；language model(abstract)；SFT(abstract)

Boosting Vulnerability Detection of LLMs via Curriculum Preference Optimization with Synthetic Reasoning Data

Xin-Cheng Wen, Yijun Yang, Cuiyun Gao, Yang Xiao, Deheng Ye

机构 * Tencent Inc.（腾讯公司） ； The Chinese University of Hong Kong（香港中文大学） ； Chinese Academy of Sciences（中国科学院） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments Accepted by ACL 2025 Findings

The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking

Yuchun Miao, Sen Zhang, Liang Ding, Yuqi Zhang, Lefei Zhang, Dacheng Tao

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments The paper has been accepted by ICML 2025

HardTests: Synthesizing High-Quality Test Cases for LLM Coding

Zhongmou He, Yee Man Choi, Kexun Zhang, Jiabao Ji, Junting Zhou, Dejia Xu, Ivan Bercovich, Aidan Zhang, Lei Li

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；post-training(abstract)

AlphaPO: Reward Shape Matters for LLM Alignment

Aman Gupta, Shao Tang, Qingquan Song, Sirou Zhu, Jiwoo Hong, Ankan Saha, Viral Gupta, Noah Lee, Eunki Kim, Siyu Zhu, Parag Agrawal, Natesh Pillai, S. Sathiya Keerthi

机构 * LinkedIn Corporation（LinkedIn公司） ； KAIST AI（KAIST人工智能） ； KAIST（韩国科学技术院） 纠错

专题命中 后训练与偏好优化 ：LLM(title)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments 26 pages, 16 figures. Accepted to ICML 2025

ExpandR: Teaching Dense Retrievers Beyond Queries with LLM Guidance

Sijia Yao, Pengcheng Huang, Zhenghao Liu, Yu Gu, Yukun Yan, Shi Yu, Ge Yu

机构 * Department of Computer Science and Technology, Northeastern University, China（东北大学计算机科学与技术系） ； Department of Computer Science and Technology, Institute for AI, Tsinghua University, China（清华大学人工智能研究院计算机科学与技术系） ； Beijing National Research Center for Information Science and Technology, China（北京信息科学与技术国家研究中心） 纠错

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Comments 16 pages, 10 tables, 5 figures

Length-Controlled Margin-Based Preference Optimization without Reference Model

Gengxu Li, Tingyu Xia, Yi Chang, Yuan Wu

机构 * School of Artificial Intelligence, Jilin University（吉林大学人工智能学院） ； Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China（知识驱动人机智能工程研究中心，教育部，中国） ； International Center of Future Science, Jilin University（未来科学国际中心，吉林大学） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments 18 pages, 3 figures, 6 tables

SCAR: Shapley Credit Assignment for More Efficient RLHF

Meng Cao, Shuyuan Zhang, Xiao-Wen Chang, Doina Precup

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；instruction tuning(abstract)