语言大模型 / LLM - arXivDaily 专题

2606.07629 2026-06-09 cs.LG cs.AI cs.CL cs.CY cs.HC 新提交 90%

Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferences

大型语言模型应学习个性化而非聚合的人类偏好

Cristina Garbacea

机构 * University of Chicago, Data Science Institute（芝加哥大学数据科学学院）

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract_cn)；分类 cs.CL、cs.AI、cs.LG

AI总结本文主张大型语言模型应学习个性化偏好而非聚合偏好，分析聚合偏好的理论局限与实证问题，提出通过有界个性化框架兼顾个体自主与集体安全。

Comments Accepted to ICML 2026

2604.26553 2026-04-30 cs.CL cs.AI cs.LG 90%

TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models

TLPO：基于令牌级别的策略优化以缓解大语言模型中的语言混淆

Jinho Choo, JunSeung Lee, Jimyeong Kim, Yeeho Song, S. K. Hong, Yeong-Dae Kwon

机构 * Samsung SDS（三星SDS）

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract_cn)；分类 cs.CL、cs.AI、cs.LG

AI总结本文提出TLPO，通过令牌级优化缓解大语言模型中的语言混淆问题，通过局部更新提升语言一致性，同时保持下游任务准确性。

Comments Accepted to the main conference of ACL 2026

2604.19485 2026-04-22 cs.LG cs.AI cs.CL 90%

EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training

EVPO：用于大语言模型后训练中自适应批评者利用的解释方差策略优化

Chengjun Pan, Shichun Liu, Jiahang Lin, Dingwei Zhu, Jiazheng Zhang, Shihan Dou, Songyang Gao, Zhenhua Han, Binghai Wang, Rui Zheng, Xuanjing Huang, Tao Gui, Yansong Feng

机构 * Peking University（北京大学）； Fudan University（复旦大学）； Shanghai Qiji Zhifeng Co., Ltd.（上海启智科技有限公司）； Shanghai AI Lab（上海人工智能实验室）

专题命中后训练与偏好优化：LLM(title,abstract)；post-training(title,abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结本文提出EVPO，通过监控批量方差并自适应切换基于批评者或批量均值估计，证明在每一步都能达到更小的方差。在四个任务中，EVPO优于PPO和GRPO。

2509.21060 2026-03-10 eess.AS 90%

Measuring Audio's Impact on Correctness: Audio-Contribution-Aware Post-Training of Large Audio Language Models

测量音频对正确性的影响：面向大音频语言模型的音频贡献意识后训练

Haolin He, Xingjian Du, Renhe Sun, Zheqi Dai, Yujia Xiao, Mingru Yang, Jiayi Zhou, Xiquan Li, Zhengxi Liu, Zining Liang, Chunyat Wu, Qianhua He, Tan Lee, Xie Chen, Wei-Long Zheng, Weiqiang Wang, Mark Plumbley, Jian Liu, Qiuqiang Kong

专题命中后训练与偏好优化：language model(title,abstract)；post-training(title,abstract)；foundation model(abstract)；SFT(abstract)

AI总结本文提出AudioMCQ数据集和Audio-Contribution Filtering方法，通过弱到强和混合到强的后训练范式，提升大音频语言模型的音频贡献意识和性能。

2504.21023 2026-03-04 cs.CL cs.AI cs.LG 90%

Param$Δ$ for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost

ParamΔ 用于直接权重混合：零成本的后训练大语言模型

Sheng Cao, Mingrui Wu, Karthik Prasad, Yuandong Tian, Zechun Liu

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；post-training(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 ParamΔ通过零成本方法实现基础模型的后训练能力迁移，提升模型性能与开发效率。

Comments Published as a conference paper at ICLR 2025

Journal ref ICLR 2025

2504.03622 2026-02-04 cs.CL cs.AI cs.LG 90%

Align to Structure: Aligning Large Language Models with Structural Information

对齐结构：将大型语言模型与结构信息对齐

Zae Myung Kim, Anand Ramachandran, Farideh Tavazoee, Joo-Kyung Kim, Oleg Rokhlenko, Dongyeop Kang

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结通过整合语言学话语框架到强化学习中，结构对齐方法提升LLMs在长文本生成和摘要任务中的连贯性和结构组织能力。

Comments Accepted to AAAI 2026 AIA

2512.13699 2025-12-17 cs.CY 90%

Us-vs-Them bias in Large Language Models

大语言模型中的‘我们 vs 他们’偏见

Tabia Tanzin Prama, Julia Witte Zimmerman, Christopher M. Danforth, Peter Sheridan Dodds

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；preference optimization(abstract)

AI总结本研究发现大语言模型在不同人设下表现出‘我们 vs 他们’偏见，通过微调和DPO方法可有效缓解这种偏见。

2511.12596 2025-11-18 cs.CL cs.AI cs.LG 90%

Group-Aware Reinforcement Learning for Output Diversity in Large Language Models

Oron Anschel, Alon Shoshan, Adam Botach, Shunit Haviv Hakimi, Asaf Gendler, Emanuel Ben Baruch, Nadav Bhonker, Igor Kviatkovsky, Manoj Aggarwal, Gerard Medioni

机构 * Amazon（亚马逊）

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments EMNLP Main 2025

2510.21090 2025-10-27 cs.CL cs.AI cs.LG 90%

Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only

Qingru Zhang, Liang Qiu, Ilgee Hong, Zhenghao Xu, Tianyi Liu, Shiyang Li, Rongzhi Zhang, Zheng Li, Lihong Li, Bing Yin, Chao Zhang, Jianshu Chen, Haoming Jiang, Tuo Zhao

机构 * Georgia Institute of Technology（佐治亚理工学院）； Amazon（亚马逊）

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；SFT(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by COLM 2025

2410.23022 2025-10-27 cs.LG cs.AI cs.CL cs.RO 90%

Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback

Qinqing Zheng, Mikael Henaff, Amy Zhang, Aditya Grover, Brandon Amos

机构 * \dagger（机构1）

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments RLC 2025

2506.18369 2025-10-13 cs.CV 90%

RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Models

Yeongtak Oh, Dohyun Chung, Juhyeon Shin, Sangha Park, Johan Barthelemy, Jisoo Mok, Sungroh Yoon

专题命中后训练与偏好优化：language model(title,abstract)；post-training(title,abstract)；large language model(abstract)；SFT(abstract)

Comments Accepted to NeurIPS 2025

2507.01915 2025-07-03 cs.CL cs.AI cs.LG 90%

Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models

Chengao Li, Hanyu Zhang, Yunkun Xu, Hongyan Xue, Xiang Ao, Qing He

机构 * Key Lab of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (CAS)（智能信息处理重点实验室，计算技术研究所，中国科学院）； State Key Lab of AI Safety（人工智能安全国家重点实验室）； University of Chinese Academy of Sciences, CAS（中国科学院大学）； Zhejiang University（浙江大学）

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 19 pages, 3 figures. Accepted by ACL 2025 (main)

2407.07880 2025-04-21 cs.LG cs.AI cs.CL 90%

Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization

Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, Xiangnan He

专题命中后训练与偏好优化：language model(title,abstract)；preference optimization(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.10596 2025-03-04 cs.CL cs.AI cs.LG 90%

Post-training an LLM for RAG? Train on Self-Generated Demonstrations

Matthew Finlayson, Ilia Kulikov, Daniel M. Bikel, Barlas Oguz, Xilun Chen, Aasish Pappu

专题命中后训练与偏好优化：LLM(title,abstract)；post-training(title)；large language model(abstract)；language model(abstract)

2502.04354 2025-02-10 cs.CL cs.AI cs.LG 90%

Reviving The Classics: Active Reward Modeling in Large Language Model Alignment

Yunyi Shen, Hao Sun, Jean-François Ton

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2412.18171 2024-12-30 cs.CR 90%

Token Highlighter: Inspecting and Mitigating Jailbreak Prompts for Large Language Models

Xiaomeng Hu, Pin-Yu Chen, Tsung-Yi Ho

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；RLHF(abstract)

Comments Accepted by AAAI 2025. Project page: https://huggingface.co/spaces/TrustSafeAI/Token-Highlighter

2412.16451 2024-12-24 cs.LG cs.AI cs.CL 90%

Correcting Large Language Model Behavior via Influence Function

Han Zhang, Zhuo Zhang, Yi Zhang, Yuanzhao Zhai, Hanyang Peng, Yu Lei, Yue Yu, Hui Wang, Bin Liang, Lin Gui, Ruifeng Xu

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；post-training(abstract)；分类 cs.CL、cs.AI、cs.LG

2411.14797 2024-11-25 cs.LG cs.AI cs.CL cs.CV 90%

Continual SFT Matches Multimodal RLHF with Negative Supervision

Ke Zhu, Yu Wang, Yanpeng Sun, Qiang Chen, Jiangjiang Liu, Gang Zhang, Jingdong Wang

专题命中后训练与偏好优化：SFT(title,abstract)；RLHF(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2405.19544 2024-11-25 cs.AI cs.CL cs.LG math.OC stat.ML 90%

One-Shot Safety Alignment for Large Language Models via Optimal Dualization

Xinmeng Huang, Shuo Li, Edgar Dobriban, Osbert Bastani, Hamed Hassani, Dongsheng Ding

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 32 pages, 6 figures, 8 tables

2305.18290 2024-07-31 cs.LG cs.AI cs.CL 90%

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn

专题命中后训练与偏好优化：language model(title,abstract)；preference optimization(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2406.02756 2024-06-06 cs.CL cs.AI cs.LG 90%

Aligning Large Language Models via Fine-grained Supervision

Dehong Xu, Liang Qiu, Minseok Kim, Faisal Ladhak, Jaeyoung Do

专题命中后训练与偏好优化：language model(title,abstract)；large language model(title)；LLM(abstract)；RLHF(abstract)

2405.21046 2024-06-03 cs.LG cs.AI cs.CL stat.ML 90%

Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF

Tengyang Xie, Dylan J. Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, Alexander Rakhlin

专题命中后训练与偏好优化：RLHF(title,abstract)；preference optimization(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2402.10958 2024-05-29 cs.CL cs.AI cs.LG 90%

Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts

Yueqin Yin, Zhendong Wang, Yi Gu, Hai Huang, Weizhu Chen, Mingyuan Zhou

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(title)；large language model(abstract)；language model(abstract)

2404.09248 2024-04-16 cs.LG cs.AI cs.CL 90%

Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts

Jing-Cheng Pang, Si-Hang Yang, Kaiyuan Li, Jiaji Zhang, Xiong-Hui Chen, Nan Tang, Yang Yu

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.AI、cs.LG

2403.09798 2024-03-18 cs.CY 90%

Comparing Rationality Between Large Language Models and Humans: Insights and Open Questions

Dana Alsagheer, Rabimba Karanjai, Nour Diallo, Weidong Shi, Yang Lu, Suha Beydoun, Qiaoning Zhang

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；RLHF(abstract)

2402.19464 2024-03-01 cs.LG cs.AI cs.CL 90%

Curiosity-driven Red-teaming for Large Language Models

Zhang-Wei Hong, Idan Shenfeld, Tsun-Hsuan Wang, Yung-Sung Chuang, Aldo Pareja, James Glass, Akash Srivastava, Pulkit Agrawal

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Published at ICLR 2024

2402.07282 2024-02-14 cs.CL cs.AI cs.LG 90%

How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?

Ryan Liu, Theodore R. Sumers, Ishita Dasgupta, Thomas L. Griffiths

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；prompting(abstract)；分类 cs.CL、cs.AI、cs.LG

2608.16553 2026-08-18 cs.CL 新提交 90%

STAGE: Controlled Objective Admission for Multi-Preference LLM Alignment

STAGE：面向多偏好大语言模型对齐的可控目标准入

Yongqi Tong, Zhenyu Zhang, Ruirui Wang, Kewei Fu, Shaoqing Lin, Sijie Dong, Jiang-Ming Yang, Xin Zhang, Jianshe Li

机构 * Ant International（蚂蚁国际）

专题命中后训练与偏好优化：LLM(title,summary_cn)；RLHF(summary_cn,abstract)；分类 cs.CL

AI总结该研究针对多偏好LLM对齐的目标准入时机问题，提出稳定性引导的STAGE控制器，通过活动集扩展与探测排序等方法，在多偏好对齐任务中取得优于基线的自动评估结果，为RLHF提供新控制变量。

2606.06586 2026-08-10 cs.CL 版本更新 90%

PolyFact: Comparing Consistency-Driven Post-training Methods for Cross-Lingual Factual Recall

通过一致性驱动的强化学习改进跨语言事实回忆

Jonathan von Rad, Louis Arts, George Burgess, Eleftheria Kolokytha, Harry O'Donnell, Ektor Oikonomidis Doumpas, Eduardo Sanchez, Yao Lu, Pontus Stenetorp

机构 * University College London（伦敦大学学院）； Centre for Artificial Intelligence（人工智能中心）

专题命中后训练与偏好优化：post-training(title,abstract)；SFT(abstract,abstract_cn)；large language model(abstract)；language model(abstract)

AI总结提出PolyFact数据集，利用GRPO强化学习方法提升大语言模型的跨语言事实回忆一致性，优于监督微调，并揭示其通过减少语言专用表示实现跨语言共享的机制。

Comments Under Review at EACL 2026

2608.06246 2026-08-07 cs.LG 新提交 90%

A Six-Dimensional Taxonomy of Post-Training Adaptation Techniques with Applications in AI Governance

用于人工智能治理的后训练适应技术的六维分类法

Fardin Afdideh, Fernando Seoane, Farhad Abtahi

专题命中后训练与偏好优化：post-training(title,abstract)；large language model(abstract)；language model(abstract)；foundation model(abstract)

AI总结本综述构建了后训练适应技术的六维分类法，梳理技术间关系，为AI治理提供术语支持，并指出该领域的开放挑战。

1. 后训练与偏好优化 4535 篇

Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferences

TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models

EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training

Measuring Audio's Impact on Correctness: Audio-Contribution-Aware Post-Training of Large Audio Language Models

Param$Δ$ for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost

Align to Structure: Aligning Large Language Models with Structural Information

Us-vs-Them bias in Large Language Models

Group-Aware Reinforcement Learning for Output Diversity in Large Language Models

Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only

Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback

RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Models

Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models

Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization

Post-training an LLM for RAG? Train on Self-Generated Demonstrations

Reviving The Classics: Active Reward Modeling in Large Language Model Alignment

Token Highlighter: Inspecting and Mitigating Jailbreak Prompts for Large Language Models

Correcting Large Language Model Behavior via Influence Function

Continual SFT Matches Multimodal RLHF with Negative Supervision

One-Shot Safety Alignment for Large Language Models via Optimal Dualization

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Aligning Large Language Models via Fine-grained Supervision

Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF

Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts

Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts

Comparing Rationality Between Large Language Models and Humans: Insights and Open Questions

Curiosity-driven Red-teaming for Large Language Models

How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?

STAGE: Controlled Objective Admission for Multi-Preference LLM Alignment

PolyFact: Comparing Consistency-Driven Post-training Methods for Cross-Lingual Factual Recall

A Six-Dimensional Taxonomy of Post-Training Adaptation Techniques with Applications in AI Governance