语言大模型 / LLM - arXivDaily 专题

2410.04834 2024-10-28 cs.CL 85%

As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss

Xin Mao, Feng-Lin Li, Huimin Xu, Wei Zhang, Wang Chen, Anh Tuan Luu

专题命中后训练与偏好优化：LLM(title,abstract)；RLHF(abstract)；preference optimization(abstract)；分类 cs.CL

Comments 20 pages, 9 figures

2407.19829 2024-10-28 cs.IR cs.AI 85%

Generative Retrieval with Preference Optimization for E-commerce Search

Mingming Li, Huimu Wang, Zuxu Chen, Guangtao Nie, Yiming Qiu, Guoyu Tang, Lin Liu, Jingwei Zhuo

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

2410.15651 2024-10-22 cs.LG 85%

Understanding and Alleviating Memory Consumption in RLHF for LLMs

Jin Zhou, Hanmei Yang, Steven, Tang, Mingcan Xiang, Hui Guan, Tongping Liu

专题命中后训练与偏好优化：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.LG

2410.15096 2024-10-22 cs.AI 85%

GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNets

Oh Joon Kwon, Daiki E. Matsunaga, Kee-Eung Kim

专题命中后训练与偏好优化：language model(title,abstract)；RLHF(abstract)；preference optimization(abstract)；分类 cs.AI

Journal ref EMNLP 2024

2406.19949 2024-10-15 cs.CL 85%

Calibrating LLMs with Preference Optimization on Thought Trees for Generating Rationale in Science Question Scoring

Jiazheng Li, Hainiu Xu, Zhaoyue Sun, Yuxiang Zhou, David West, Cesare Aloisi, Yulan He

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments Accepted at EMNLP 2024

2406.15951 2024-10-14 cs.CL 85%

Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration

Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, Yulia Tsvetkov

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments EMNLP 2024

2409.08845 2024-09-16 cs.CL 85%

AIPO: Improving Training Objective for Iterative Preference Optimization

Yaojie Shen, Xinyao Wang, Yulei Niu, Ying Zhou, Lexin Tang, Libo Zhang, Fan Chen, Longyin Wen

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

2407.05013 2024-07-09 cs.CL 85%

Progress or Regress? Self-Improvement Reversal in Post-training

Ting Wu, Xuefeng Li, Pengfei Liu

专题命中后训练与偏好优化：post-training(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

2402.16029 2024-07-04 cs.CL 85%

GraphWiz: An Instruction-Following Language Model for Graph Problems

Nuo Chen, Yuhan Li, Jianheng Tang, Jia Li

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；preference optimization(abstract)；分类 cs.CL

Comments 27pages, 15 tables

2406.17092 2024-06-26 cs.CR cs.AI 85%

BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Models

Yi Zeng, Weiyu Sun, Tran Ngoc Huynh, Dawn Song, Bo Li, Ruoxi Jia

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；RLHF(abstract)；分类 cs.AI

2406.08842 2024-06-14 cs.CL 85%

ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions

Xu Zhang, Xunjian Yin, Xiaojun Wan

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；preference optimization(abstract)；分类 cs.CL

2406.08657 2024-06-14 cs.CL 85%

Mistral-C2F: Coarse to Fine Actor for Analytical and Reasoning Enhancement in RLHF and Effective-Merged LLMs

Chen Zheng, Ke Sun, Xun Zhou

专题命中后训练与偏好优化：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

2403.13787 2024-06-11 cs.LG 85%

RewardBench: Evaluating Reward Models for Language Modeling

Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi

专题命中后训练与偏好优化：language model(title,abstract)；RLHF(abstract)；preference optimization(abstract)；分类 cs.LG

Comments 44 pages, 19 figures, 12 tables

2405.14103 2024-05-24 cs.LG 85%

Online Self-Preferring Language Models

Yuanzhao Zhai, Zhuo Zhang, Kele Xu, Hanyang Peng, Yue Yu, Dawei Feng, Cheng Yang, Bo Ding, Huaimin Wang

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；RLHF(abstract)；分类 cs.LG

Comments 20 pages, 9 figures

2404.19409 2024-05-01 cs.CL 85%

Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning

Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, Olivier Pietquin

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

2311.08380 2024-04-15 cs.CL 85%

Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding

Guangyu Yang, Jinghong Chen, Weizhe Lin, Bill Byrne

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments To appear at NAACL 2024

2311.16839 2024-02-07 cs.CV cs.CL 85%

Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization

Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, Conghui He

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments Project Website: https://opendatalab.github.io/HA-DPO, Code: https://github.com/opendatalab/HA-DPO

2401.10580 2024-01-22 cs.CL 85%

PHOENIX: Open-Source Language Adaption for Direct Preference Optimization

Matthias Uhlig, Sigurd Schacht, Sudarshan Kamath Barkur

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

2401.07181 2024-01-17 cs.LG 85%

Reinforcement Learning from LLM Feedback to Counteract Goal Misgeneralization

Houda Nait El Barj, Theophile Sautory

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.LG

2305.14975 2023-10-25 cs.CL 85%

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, Christopher D. Manning

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；RLHF(abstract)；分类 cs.CL

Comments EMNLP 2023 Camera Ready

2309.14525 2023-09-27 cs.CV cs.CL 85%

Aligning Large Multimodal Models with Factually Augmented RLHF

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, Trevor Darrell

专题命中后训练与偏好优化：RLHF(title,abstract)；language model(abstract)；instruction tuning(abstract)；分类 cs.CL

Comments Preprint

2309.13308 2023-09-26 cs.CL 85%

Calibrating LLM-Based Evaluator

Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments 22 pages,11 figures

2309.09055 2023-09-19 cs.CL 85%

Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHF

Simeng Sun, Dhawal Gupta, Mohit Iyyer

专题命中后训练与偏好优化：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

2608.21376 2026-08-25 cs.CL cs.AI 新提交 85%

On the Role of Citations in Preference Data

引用在偏好数据中的作用

Yu Hou, Hal Daumé, Rachel Rudinger, William Walden

机构 * University of Maryland（马里兰大学）； Johns Hopkins University（约翰斯·霍普金斯大学）

专题命中后训练与偏好优化：LLM(summary_cn,abstract)；post-training(abstract)；分类 cs.CL、cs.AI

AI总结本文在科学问答场景下，通过混合效应模型研究引用对人类与四个开源LLM偏好的影响，发现人类偏好多样且数量少的引用，LLM也有相关偏好，还探讨了对偏好数据收集的启示。

2605.00155 2026-08-11 cs.LG cs.CL math.OC stat.ML 版本更新 85%

Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback

Wasserstein分布鲁棒遗憾优化用于人类反馈的强化学习

Yikai Wang, Shang Liu, Jose Blanchet

机构 * Department of Statistics and Operations Research, University of North Carolina（统计与运筹学系，北卡罗来纳大学）； Imperial Business School, Imperial College London（帝国理工学院伦敦商学院）； Department of Management Science and Engineering, Stanford University（管理科学与工程系，斯坦福大学）

专题命中后训练与偏好优化：RLHF(abstract,abstract_cn)；large language model(abstract)；language model(abstract)；post-training(abstract)

AI总结本文提出Wasserstein分布鲁棒遗憾优化（DRRO）用于强化学习从人类反馈，通过简单分配模型研究提示问题，展示在ℓ1-地面成本Wasserstein模糊集下，内最坏遗憾有精确解，最优策略具有水填充结构，从而实现高效政策梯度算法。

2606.21337 2026-08-04 cs.LG cs.AI 版本更新 85%

DataClaw0: Agentic Tailoring Multimodal Data from Raw Streams

DataClaw0: 从原始流中智能定制多模态数据

Cong Wan, Zeyu Guo, Zijian Cai, Jiangyang Li, SongLin Dong, Lin Peng, Xiangyang Luo, Zhiheng Ma, Yihong Gong

机构 * University of Chinese Academy of Sciences（中国科学院大学）； Shenzhen University of Advanced Technology（深圳理工大学）； Tsinghua University（清华大学）

专题命中后训练与偏好优化：SFT(summary_cn,abstract_cn)；language model(abstract)；post-training(abstract)；分类 cs.AI、cs.LG

AI总结提出DataClaw0模型，通过两阶段流水线将生成语义合成锚定于确定性事实锚点，结合SFT与GRPO实现复杂数据精炼意图的对齐，首个数据精炼基准验证其高效性。

Comments add base model: Qwen3.5-27B

2607.03957 2026-07-29 cs.CL cs.AI 版本更新 85%

NormWorlds-CF: Solver-Verified Counterfactual Normative Reasoning with Metamorphic-Relation GRPO

NormWorlds-CF：使用变质关系GRPO进行求解器验证的反事实规范推理

Xinqi Zhang

机构 * Tsinghua University（清华大学）

专题命中后训练与偏好优化：SFT(abstract,abstract_cn)；LLM(abstract)；language model(abstract)；post-training(abstract)

AI总结介绍NormWorlds-CF用于可执行规则世界的反事实规范推理，其求解器能产生多种结果用于监督评估。通过实验对比不同奖励机制对任务的影响，显示验证的反事实结构可影响训练后表现。

2607.18163 2026-07-21 cs.LG cs.AI 新提交 85%

OR Else: A Differentiable Trust Region for Policy Optimization

OR 否则：用于策略优化的可微信赖域

Chinmay Rane, Kanishka Tyagi, Michael Manry

机构 * Quantiphi Inc（昆蒂菲公司）； Self Machines Inc（自机器公司）； The University of Texas at Arlington（德克萨斯大学阿灵顿分校）

专题命中后训练与偏好优化：RLHF(abstract,abstract_cn)；large language model(abstract)；language model(abstract)；post-training(abstract)

AI总结研究 PPO 和 GRPO 中裁剪代理目标导数突变问题，提出用输出重置（OR）规则优化。通过对比实验，在广义优势估计下 PPO-OR 有更高奖励模型得分，组相对优势下 GRPO-OR 虽平均得分未升但差异更小，OR 改变了优化行为但奖励效果有别。

Comments 22 pages, 5 figures

2606.15306 2026-06-16 cs.LG cs.AI 新提交 85%

LatentGym: A Testbed For Cross-Task Experiential Learning With Controllable Latent Structure

LatentGym: 具有可控潜在结构的跨任务经验学习测试平台

Daksh Mittal, Tommaso Castellani, Thomson Yen, Naimeng Ye, Fangyu Wu, Minghui Chen, Tiffany Cai, Emmanouil Koukoumidis, William Zeng, Hongseok Namkoong

机构 * Columbia University（哥伦比亚大学）； Oumi Blog | Code | Models（Oumi博客 | 代码 | 模型）

专题命中后训练与偏好优化：LLM(summary_cn,abstract)；post-training(abstract)；分类 cs.AI、cs.LG

AI总结提出LatentGym测试平台，通过可控潜在变量分离探索与利用，研究LLM代理在跨任务序列中的适应性学习机制。

Comments 61 pages

2602.01023 2026-06-10 cs.IR cs.AI cs.LG 85%

Unifying Ranking and Generation in Query Auto-Completion via Retrieval-Augmented Generation and Multi-Objective Alignment

通过检索增强生成和多目标对齐统一查询自动补全中的排序与生成

Kai Yuan, Anthony Zheng, Jia Hu, Divyanshu Sheth, Hemanth Velaga, Kylee Kim, Matteo Guarrera, Besim Avci, Jianhua Li, Xuetao Yin, Rajyashree Mukherjee, Sean Suchter

机构 * Apple（苹果公司）； UC Berkeley（加州大学伯克利分校）

专题命中后训练与偏好优化：LLM(abstract,abstract_cn)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

AI总结提出一个统一框架，通过检索增强生成（RAG）和多目标直接偏好优化（DPO）将查询自动补全重构为端到端列表生成，解决传统流水线长尾覆盖不足和生成方法幻觉风险的问题，并在大规模商业搜索平台上验证了有效性。

Comments 11 pages, 4 figures

Journal ref Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD '26), August 09--13, 2026, Jeju Island, Republic of Korea

1. 后训练与偏好优化 4556 篇

As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss

Generative Retrieval with Preference Optimization for E-commerce Search

Understanding and Alleviating Memory Consumption in RLHF for LLMs

GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNets

Calibrating LLMs with Preference Optimization on Thought Trees for Generating Rationale in Science Question Scoring

Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration

AIPO: Improving Training Objective for Iterative Preference Optimization

Progress or Regress? Self-Improvement Reversal in Post-training

GraphWiz: An Instruction-Following Language Model for Graph Problems

BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Models

ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions

Mistral-C2F: Coarse to Fine Actor for Analytical and Reasoning Enhancement in RLHF and Effective-Merged LLMs

RewardBench: Evaluating Reward Models for Language Modeling

Online Self-Preferring Language Models

Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning

Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding

Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization

PHOENIX: Open-Source Language Adaption for Direct Preference Optimization

Reinforcement Learning from LLM Feedback to Counteract Goal Misgeneralization

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

Aligning Large Multimodal Models with Factually Augmented RLHF

Calibrating LLM-Based Evaluator

Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHF

On the Role of Citations in Preference Data

Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback

DataClaw0: Agentic Tailoring Multimodal Data from Raw Streams

NormWorlds-CF: Solver-Verified Counterfactual Normative Reasoning with Metamorphic-Relation GRPO

OR Else: A Differentiable Trust Region for Policy Optimization

LatentGym: A Testbed For Cross-Task Experiential Learning With Controllable Latent Structure

Unifying Ranking and Generation in Query Auto-Completion via Retrieval-Augmented Generation and Multi-Objective Alignment