语言大模型 / LLM - arXivDaily 专题

2409.15360 2024-10-17 cs.LG cs.AI cs.CL 88%

Reward-Robust RLHF in LLMs

Yuzi Yan, Xingzhou Lou, Jialian Li, Yiping Zhang, Jian Xie, Chao Yu, Yu Wang, Dong Yan, Yuan Shen

专题命中后训练与偏好优化：RLHF(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

2312.11819 2024-10-15 cs.LG cs.AI cs.CL 88%

An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training

Youshao Xiao, Zhenglei Zhou, Fagui Mao, Weichang Wu, Shangchun Zhao, Lin Ju, Lei Liang, Xiaolu Zhang, Jun Zhou

专题命中后训练与偏好优化：RLHF(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

2404.05868 2024-10-14 cs.LG cs.AI cs.CL stat.ML 88%

Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning

Ruiqi Zhang, Licong Lin, Yu Bai, Song Mei

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

2409.10164 2024-09-17 cs.LG cs.AI cs.CL 88%

Quantile Regression for Distributional Reward Models in RLHF

Nicolai Dorka

专题命中后训练与偏好优化：RLHF(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

2309.00267 2024-09-04 cs.CL cs.AI cs.LG 88%

RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, Sushant Prakash

专题命中后训练与偏好优化：RLHF(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments Presented at ICML 2024

Journal ref Proceedings of the 41st International Conference on Machine Learning, PMLR 235:26874-26901, 2024

2405.20053 2024-05-31 cs.CL cs.AI cs.LG 88%

Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads

Avelina Asada Hadji-Kyriacou, Ognjen Arandjelovic

专题命中后训练与偏好优化：language model(title,abstract)；SFT(abstract)；RLHF(abstract)；preference optimization(abstract)

2404.03715 2024-04-08 cs.LG cs.AI cs.CL 88%

Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences

Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, Tengyang Xie

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；post-training(abstract)；RLHF(abstract)

2403.06754 2024-03-19 cs.CL cs.AI cs.LG 88%

ALaRM: Align Language Models via Hierarchical Rewards Modeling

Yuhang Lai, Siyuan Wang, Shujun Liu, Xuanjing Huang, Zhongyu Wei

专题命中后训练与偏好优化：language model(title,abstract)；LLM(abstract)；large language model(abstract)；RLHF(abstract)

Comments 15 pages, 6 figures

2304.06767 2023-12-04 cs.LG cs.AI cs.CL cs.CV stat.ML 88%

RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment

Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, Tong Zhang

专题命中后训练与偏好优化：foundation model(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments 29 pages, 12 figures, Published in Transactions on Machine Learning Research (TMLR)

2512.08786 2025-12-17 cs.CL cs.AI 88%

A Systematic Evaluation of Preference Aggregation in Federated RLHF for Pluralistic Alignment of LLMs

对联邦RLHF中偏好聚合的系统评估：为LLM的多元化对齐

Mahmoud Srewa, Tianyu Zhao, Salma Elmalaki

机构 * Department of Electrical Engineering and Computer Science University of California, Irvine（电气工程与计算机科学系加州大学伊文斯分校）

专题命中后训练与偏好优化：RLHF(title,abstract)；LLM(abstract,comments)；large language model(abstract)；language model(abstract)

AI总结本文提出了一种自适应偏好聚合方法，通过动态调整权重提升联邦RLHF中LLM的公平性与对齐性能。

Comments This paper is accepted at the NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle

2312.11456 2024-05-02 cs.LG cs.AI stat.ML 88%

Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint

Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, Tong Zhang

专题命中后训练与偏好优化：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Comments 53 pages; theoretical study and algorithmic design of iterative RLHF and DPO

2608.15504 2026-08-18 cs.LG stat.ML 新提交 88%

PERO: Efficient Robust Post-Training Foundation Models for Encrypted Traffic Classification

PERO：面向加密流量分类的高效鲁棒预训练后 foundation 模型

Wumei Du, Jiarong Wen, Kaiyu Zhang, Zi Yang, Yiqin Lv, Longfei Zhang, Dong Liang, Zheng Xie

机构 * National University of Defense Technology（国防科技大学）

专题命中后训练与偏好优化：foundation model(title,abstract)；post-training(title,abstract)；分类 cs.LG

AI总结针对加密流量分类中鲁棒优化成本高的问题，提出PERO框架，通过轻量代理估计风险并选择高风险样本更新模型，在保持性能的同时降低了计算与内存成本。

Comments 16 pages, 6 figures, 6 tables, conference

2607.22614 2026-08-03 cs.AI 版本更新 88%

DynaResize: Runtime GPU Reallocation for Disaggregated LLM Post-Training

DynaResize：用于分布式语言模型训练后运行时的GPU重新分配

Hanlin Du, Zhiyuan Yan, Yungang Bao, Sa wang

机构 * SKLP, Institute of Computing Technology, CAS（中国科学院计算技术研究所智能处理器研究中心）； Bytedance（字节跳动）

专题命中后训练与偏好优化：LLM(title,abstract)；post-training(title,abstract)；分类 cs.AI

AI总结研究基于强化学习的语言模型训练后GPU资源分配问题，提出DynaResize运行时GPU重新分配系统，通过动态切换GPU平衡阶段执行时间，分解操作并去除关键路径非关键工作，实验显示可提高吞吐量、减少执行时间并隐藏部分开销。

2606.25328 2026-06-25 cs.SD cs.AI 新提交 88%

Supervised Post-training of Speech Foundation Models for Robust Adaptation in Speech Deepfake Detection

语音深度伪造检测中鲁棒适应的语音基础模型监督后训练

Zihan Pan, Sailor Hardik, Jinyang Wu

机构 * Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR)（资讯通信研究院（I2R），新加坡科技研究局（A*STAR））

专题命中后训练与偏好优化：foundation model(title,abstract)；post-training(title,abstract)；分类 cs.AI

AI总结提出混合帧后训练策略，通过帧级监督学习局部不一致性，在ASVspoof5上实现单模型EER 4.50%，在ASVspoof2021上LA与DF的EER差距仅0.16%。

2606.17967 2026-06-25 cs.CL 新提交 88%

Learning task-specific subspaces via interventional post-training of speech foundation models

通过语音基础模型的干预后训练学习任务特定子空间

Jack Cox, Jon Barker

机构 * University of Sheffield（谢菲尔德大学）

专题命中后训练与偏好优化：foundation model(title,abstract)；post-training(title,abstract)；分类 cs.CL

AI总结提出一种干预对比学习后训练方法，将语音基础模型的纠缠表示分解为内容和说话人子空间，提升说话人验证和关键词识别性能。

Comments Accepted to Interspeech 2026; 6 pages (4 main body), 2 figures

2606.21458 2026-06-23 cs.LG 新提交 88%

Post-Training Speech Enhancement Language Models with Perceptual Rewards

基于感知奖励的语音增强语言模型后训练

Frédéric Berdoz, Luca A. Lanzendörfer, Antonis Asonitis, Roger Wattenhofer

机构 * ETH Zürich（苏黎世联邦理工学院）

专题命中后训练与偏好优化：language model(title,abstract)；post-training(title,abstract)；分类 cs.LG

AI总结提出使用组序列策略优化（GSPO）结合多指标感知奖励对自回归语音增强语言模型进行后训练，直接优化非可微质量指标，在DNS2020基准上达到最优，且多奖励优化优于单指标变体。

Comments Accepted at Interspeech 2026

2605.20408 2026-05-21 cs.LG 88%

Spectral Souping: A Unified Framework for Online Preference Alignment

谱汤：一种在线偏好对齐的统一框架

Yinlam Chow, Guy Tennenholtz, Ted Yun, James Harrison, Arthur Gretton, Andre Barreto, Bo Dai

机构 * Google DeepMind（谷歌DeepMind）； Google Research（谷歌研究）

专题命中后训练与偏好优化：LLM(summary_cn,abstract_cn)；RLHF(abstract,abstract_cn)；large language model(abstract)；language model(abstract)

AI总结本文提出了一种统一的在线偏好对齐框架Spectral Souping，通过发现LLM中的通用谱表示，实现了高效的模型合并，从而在不需昂贵在线重训练的情况下快速适应个体用户偏好。

2605.16842 2026-05-19 cs.AI 88%

Sketch Then Paint: Hierarchical Reinforcement Learning for Diffusion Multi-Modal Large Language Models

草图然后绘画：用于扩散多模态大语言模型的分层强化学习

Siqi Luo, Jianghan Shen, Yi Xin, Huayu Zheng, Haoxing Chen, Yan Tai, Yue Li, Junjun He, Yihao Liu, Guangtao Zhai, Yuewen Cao, Xiaohong Liu

机构 * Shanghai Jiao Tong University（上海交通大学）； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）； Nanjing University（南京大学）； Shanghai Innovation Institute（上海创新研究院）； Peking University（北京大学）

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；分类 cs.AI

AI总结本文提出了一种分层强化学习方法HT-GRPO，通过Sketch-Then-Paint训练方案和分层信用分配机制，解决扩散多模态大语言模型在强化学习优化中的关键问题，提升图像质量和审美效果。

2602.08813 2026-05-13 cs.LG 88%

Robust Policy Optimization to Prevent Catastrophic Forgetting

鲁棒策略优化以防止灾难性遗忘

Mahdi Sabbaghi, George Pappas, Adel Javanmard, Hamed Hassani

机构 * University of Pennsylvania（宾夕法尼亚大学）； University of Southern California（南加州大学）

专题命中后训练与偏好优化：SFT(abstract,abstract_cn)；RLHF(abstract,abstract_cn)；large language model(abstract)；language model(abstract)

AI总结本文提出FRPO框架，通过优化当前策略及下游适应可达的KL限制邻域内策略，提升鲁棒性，减少安全性能退化，同时保持下游任务表现。

2602.11181 2026-05-12 cs.CL 88%

Code Mixologist : A Practitioner's Guide to Building Code-Mixed LLMs

代码调酒师：构建代码混合LLM的从业者指南

Himanshu Gupta, Pratik Jayarao, Chaitanya Dwivedi, Neeraj Varshney

机构 * Arizona State University（亚利桑那州立大学）； Carnegie Mellon University（卡内基梅隆大学）

专题命中后训练与偏好优化：LLM(title_cn,abstract_cn)；large language model(abstract)；language model(abstract)；post-training(abstract)

AI总结本文探讨了代码混合和切换在大语言模型中的挑战，提出统一的分类体系和实用指南，涵盖数据、建模和评估方法，分析现有评估实践并讨论安全问题。

Comments 8 pages main paper, 13 pages total

2602.00986 2026-05-12 cs.CL 88%

Sparse Reward Subsystem in Large Language Models

大语言模型中的稀疏奖励子系统

Guowei Xu, Mert Yuksekgonul, James Zou

机构 * Tsinghua University（清华大学）； Stanford University（斯坦福大学）

专题命中后训练与偏好优化：large language model(title)；language model(title)；LLM(abstract,abstract_cn)；分类 cs.CL

AI总结研究揭示大语言模型隐藏状态中稀疏神经元构成奖励子系统，识别价值神经元和多巴胺神经元，用于预测模型置信度和指导推理过程。

2504.21228 2026-05-12 cs.CR cs.AI 88%

CachePrune: Teaching LLMs What Not to Follow via KV-Cache Editing

CachePrune: 通过KV缓存编辑教LLMs不遵循指令

Rui Wang, Junda Wu, Yu Xia, Tong Yu, Ruiyi Zhang, Ryan Rossi, Subrata Mitra, Lina Yao, Julian McAuley

机构 * Adobe Research（Adobe研究院）； University of California San Diego（加州大学圣地亚哥分校）； University of New South Wales（新南威尔士大学）

专题命中后训练与偏好优化：LLM(summary_cn,abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

AI总结 CachePrune通过KV缓存编辑识别并修剪指令跟随相关神经元，降低间接提示注入攻击成功率，同时保持LLM执行用户指令的能力。

2604.13833 2026-04-17 cs.CL 88%

Robust Reward Modeling for Large Language Models via Causal Decomposition

通过因果分解实现大型语言模型的鲁棒奖励建模

Yunsheng Lu, Zijiang Yang, Licheng Pan, Zhixuan Chu

机构 * College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）； Department of Statistics, University of Chicago（芝加哥大学统计系）

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL

AI总结本文提出通过因果分解学习解码器，以增强奖励模型对提示意图的建模，从而提升在数学、帮助性和安全性的基准测试中的表现。

2603.18806 2026-04-14 cs.AI 88%

dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Models

dTRPO：扩散大语言模型策略优化中的轨迹缩减

Wenxuan Zhang, Lemeng Wu, Changsheng Zhao, Ernie Chang, Mingchen Zhuge, Zechun Liu, Andy Su, Hanxian Huang, Jun Chen, Chong Zhou, Raghuraman Krishnamoorthi, Vikas Chandra, Mohamed Elhoseiny, Wei Wen

机构 * Meta AI

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；分类 cs.AI

AI总结本文提出dTRPO，通过减少轨迹概率计算成本提升扩散大语言模型的策略优化效果，实验显示在STEM、编程和指令遵循任务中性能提升显著。

2604.01597 2026-04-03 cs.LG 88%

Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training

从正确的回放中学习：基于PPO的LLM后训练的数据归因

Dong Shu, Denghui Zhang, Jessica Hullman

机构 * Northwestern University（西北大学）； Stevens Institute of Technology（史蒂文斯理工学院）

专题命中后训练与偏好优化：post-training(title,abstract)；LLM(title)；SFT(abstract)；分类 cs.LG

AI总结本文提出Influence-Guided PPO框架，通过计算影响分数筛选反向齐梯度的回放片段，提升PPO后训练效率并减少不忠实的推理。

2502.20795 2026-03-23 cs.CL 88%

Test-Time Alignment for Large Language Models via Textual Model Predictive Control

通过文本模型预测控制对大型语言模型进行测试时对齐

Kuang-Da Wang, Teng-Ruei Chen, Yu Heng Hung, Guo-Xun Ko, Shuoyang Ding, Yueh-Hua Wu, Yu-Chiang Frank Wang, Chao-Han Huck Yang, Wen-Chih Peng, Ping-Chun Hsieh

机构 * National Yang Ming Chiao Tung University ； NVIDIA ； National Taiwan University

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL

AI总结本文提出TMPC方法，通过文本模型预测控制解决测试时对齐问题，通过分层强化学习原理改进生成过程，提升不同任务的性能。

Comments Accepted for ICLR 2026. Project page: https://rl-bandits-lab.github.io/TMPC/

2603.00152 2026-03-06 cs.CV cs.AI 88%

Dr. Seg: Revisiting GRPO Training for Visual Large Language Models through Perception-Oriented Design

Dr. Seg: 通过感知导向设计重新审视GRPO训练用于视觉大语言模型

Haoxiang Sun, Tao Wang, Chenwei Tang, Li Yuan, Jiancheng Lv

机构 * College of Computer Science, Sichuan University（四川大学计算机科学学院）； School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；分类 cs.AI

AI总结 Dr.Seg通过感知导向设计重新审视GRPO训练，提出简单框架提升视觉任务性能

2602.10986 2026-02-12 cs.LG 88%

TVCACHE: A Stateful Tool-Value Cache for Post-Training LLM Agents

TVCACHE: 一种具有状态的工具-价值缓存用于训练后LLM代理

Abhishek Vijaya Kumar, Bhaskar Kataria, Byungsoo Oh, Emaad Manzoor, Rachee Singh

机构 * cornell（康奈尔大学）

专题命中后训练与偏好优化：LLM(title,abstract)；post-training(title,abstract)；分类 cs.LG

AI总结 TVCACHE通过维护工具调用序列树和最长前缀匹配，实现LLM代理训练后的高效缓存，提高缓存命中率并减少工具调用时间。

Comments Abhishek Vijaya Kumar and Bhaskar Kataria have equal contribution

2601.17918 2026-01-27 cs.CV cs.CL 88%

Benchmarking Direct Preference Optimization for Medical Large Vision-Language Models

医疗大视觉-语言模型的直接偏好优化基准测试

Dain Kim, Jiwoo Lee, Jaehoon Yun, Yong Hoe Koo, Qingyu Chen, Hyunjae Kim, Jaewoo Kang

机构 * Korea University（韩国大学）； AIGEN Sciences（AIGEN公司）； Hanyang University College of Medicine（翰林大学医学院）； Asan Medical Center, University of Ulsan College of Medicine（釜山大学医学院阿桑医院）； Yale University（耶鲁大学）

专题命中后训练与偏好优化：language model(title,abstract)；preference optimization(title,abstract)；分类 cs.CL

AI总结本文评估了医疗领域中多种DPO变体，发现其在视觉问答任务中存在性能不一致和视觉误解问题，并提出针对性策略提升3.6%。

Comments EACL 2026 (Findings)

2601.10532 2026-01-19 cs.CL 88%

PERM: Psychology-grounded Empathetic Reward Modeling for Large Language Models

PERM：基于心理学的共情奖励建模用于大语言模型

Chengbing Wang, Wuqiang Zheng, Yang Zhang, Fengbin Zhu, Junyi Cheng, Yi Xie, Wenjie Wang, Fuli Feng

机构 * University of Science and Technology of China（中国科学技术大学）； National University of Singapore（新加坡国立大学）； Huawei Technologies（华为技术）

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL

AI总结 PERM通过双向分解和旁观者视角提升大语言模型的共情能力，实验显示其在同理心基准和对话数据集上表现优异，用户偏好率达70%。

1. 后训练与偏好优化 4541 篇

Reward-Robust RLHF in LLMs

An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training

Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning

Quantile Regression for Distributional Reward Models in RLHF

RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads

Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences

ALaRM: Align Language Models via Hierarchical Rewards Modeling

RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment

A Systematic Evaluation of Preference Aggregation in Federated RLHF for Pluralistic Alignment of LLMs

Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint

PERO: Efficient Robust Post-Training Foundation Models for Encrypted Traffic Classification

DynaResize: Runtime GPU Reallocation for Disaggregated LLM Post-Training

Supervised Post-training of Speech Foundation Models for Robust Adaptation in Speech Deepfake Detection

Learning task-specific subspaces via interventional post-training of speech foundation models

Post-Training Speech Enhancement Language Models with Perceptual Rewards

Spectral Souping: A Unified Framework for Online Preference Alignment

Sketch Then Paint: Hierarchical Reinforcement Learning for Diffusion Multi-Modal Large Language Models

Robust Policy Optimization to Prevent Catastrophic Forgetting

Code Mixologist : A Practitioner's Guide to Building Code-Mixed LLMs

Sparse Reward Subsystem in Large Language Models

CachePrune: Teaching LLMs What Not to Follow via KV-Cache Editing

Robust Reward Modeling for Large Language Models via Causal Decomposition

dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Models

Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training

Test-Time Alignment for Large Language Models via Textual Model Predictive Control

Dr. Seg: Revisiting GRPO Training for Visual Large Language Models through Perception-Oriented Design

TVCACHE: A Stateful Tool-Value Cache for Post-Training LLM Agents

Benchmarking Direct Preference Optimization for Medical Large Vision-Language Models

PERM: Psychology-grounded Empathetic Reward Modeling for Large Language Models