语言大模型 / LLM - arXivDaily 专题

2503.00018 2025-03-04 cs.CL cs.AI 88%

Eeyore: Realistic Depression Simulation via Supervised and Preference Optimization

Siyang Liu, Bianca Brie, Wenda Li, Laura Biester, Andrew Lee, James Pennebaker, Rada Mihalcea

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

2501.01821 2025-02-28 cs.AI cs.CL 88%

SDPO: Segment-Level Direct Preference Optimization for Social Agents

Aobo Kong, Wentao Ma, Shiwan Zhao, Yongbin Li, Yuchuan Wu, Ke Wang, Xiaoqian Liu, Qicheng Li, Yong Qin, Fei Huang

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

2502.18532 2025-02-27 cs.AI cs.LG 88%

CuDIP: Enhancing Theorem Proving in LLMs via Curriculum Learning-based Direct Preference Optimization

Shuming Shi, Ruobing Zuo, Gaolei He, Jianlin Wang, Chenyang Xu, Zhengfeng Yang

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

2502.10993 2025-02-18 cs.CL cs.LG 88%

RoseRAG: Robust Retrieval-augmented Generation with Small-scale LLMs via Margin-aware Preference Optimization

Tianci Liu, Haoxiang Jiang, Tianze Wang, Ran Xu, Yue Yu, Linjun Zhang, Tuo Zhao, Haoyu Wang

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

2502.00203 2025-02-11 cs.LG cs.CL 88%

Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment

Shengyang Sun, Yian Zhang, Alexander Bukharin, David Mosallanezhad, Jiaqi Zeng, Soumye Singhal, Gerald Shen, Adithya Renduchintala, Tugrul Konuk, Yi Dong, Zhilin Wang, Dmitry Chichkov, Olivier Delalleau, Oleksii Kuchaiev

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments 8 pages, 4 figures; update author names

2404.01054 2025-01-30 cs.CL cs.AI 88%

Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment

Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, Kenshi Abe

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；RLHF(abstract)；preference optimization(abstract)

Comments NAACL 2025

2501.03486 2025-01-08 cs.LG cs.AI 88%

Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment

Prashant Trivedi, Souradip Chakraborty, Avinash Reddy, Vaneet Aggarwal, Amrit Singh Bedi, George K. Atia

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments 27 pages, Accepted in AAAI 2025

2409.06411 2024-12-02 cs.LG cs.CL 88%

Length Desensitization in Direct Preference Optimization

Wei Liu, Yang Bai, Chengcheng Han, Rongxiang Weng, Jun Xu, Xuezhi Cao, Jingang Wang, Xunliang Cai

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments 21 pages, 9 figures

2410.22304 2024-10-30 cs.CL cs.LG 88%

Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learning

Yihe Deng, Paul Mineiro

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Comments 5 pages, 4 figures, 1 table

2405.15230 2024-10-30 cs.AI cs.LG 88%

$i$REPO: $i$mplicit Reward Pairwise Difference based Empirical Preference Optimization

Long Tan Le, Han Shu, Tung-Anh Nguyen, Choong Seon Hong, Nguyen H. Tran

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments Under Review

2410.20298 2024-10-29 cs.CL cs.AI 88%

Learning from Response not Preference: A Stackelberg Approach for LLM Detoxification using Non-parallel Data

Xinhong Xie, Tao Li, Quanyan Zhu

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

2410.20187 2024-10-29 cs.LG cs.AI stat.ML 88%

Uncertainty-Penalized Direct Preference Optimization

Sam Houliston, Alizée Pace, Alexander Immer, Gunnar Rätsch

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments Accepted at the NeurIPS 2024 FITML Workshop

2406.11715 2024-10-28 cs.LG cs.CL cs.SE 88%

Measuring memorization in RLHF for code completion

Aneesh Pappu, Billy Porter, Ilia Shumailov, Jamie Hayes

专题命中后训练与偏好优化：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

2409.03650 2024-10-04 cs.LG cs.CL 88%

On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization

Yong Lin, Skyler Seto, Maartje ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang, Yizhe Zhang, Chen Huang, Tong Zhang

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract)；language model(abstract)；RLHF(abstract)

Comments 12 pages, 8 tables, 3 figures; Paper Accepted at EMNLP Findings 2024

2403.19159 2024-09-10 cs.CL cs.LG 88%

Disentangling Length from Quality in Direct Preference Optimization

Ryan Park, Rafael Rafailov, Stefano Ermon, Chelsea Finn

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

2407.12847 2024-07-19 cs.CL cs.AI cs.HC 88%

Aligning Model Evaluations with Human Preferences: Mitigating Token Count Bias in Language Model Assessments

Roland Daynauth, Jason Mars

专题命中后训练与偏好优化：language model(title,abstract)；LLM(abstract)；large language model(abstract)；small language model(abstract)

2402.15018 2024-06-10 cs.CL cs.CY cs.LG 88%

Unintended Impacts of LLM Alignment on Global Representation

Michael J. Ryan, William Held, Diyi Yang

专题命中后训练与偏好优化：LLM(title)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments Accepted to ACL 2024 main conference

2402.06700 2024-06-07 cs.LG cs.AI 88%

Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement

Muning Wen, Junwei Liao, Cheng Deng, Jun Wang, Weinan Zhang, Ying Wen

专题命中后训练与偏好优化：language agent(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

2405.21040 2024-06-03 cs.CL cs.AI 88%

Direct Alignment of Language Models via Quality-Aware Self-Refinement

Runsheng Yu, Yong Wang, Xiaoqi Jiao, Youzhi Zhang, James T. Kwok

专题命中后训练与偏好优化：language model(title,abstract)；LLM(abstract)；large language model(abstract)；RLHF(abstract)

2310.16836 2024-04-30 cs.CL cs.AI cs.AR cs.CV 88%

LLM-FP4: 4-Bit Floating-Point Quantized Transformers

Shih-yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, Kwang-Ting Cheng

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；post-training(abstract)

Comments EMNLP 2023 Main Conference

Journal ref Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

2403.00199 2024-04-22 cs.CL cs.CY cs.LG 88%

Improving Socratic Question Generation using Data Augmentation and Preference Optimization

Nischal Ashok Kumar, Andrew Lan

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

Comments Published at the 19th BEA Workshop co-located with NAACL-2024

2312.02554 2024-02-27 cs.LG cs.CL 88%

ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference

Tianchi Cai, Xierui Song, Jiyan Jiang, Fei Teng, Jinjie Gu, Guannan Zhang

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；RLHF(abstract)；preference optimization(abstract)

2311.12908 2023-11-23 cs.CV cs.AI cs.GR cs.LG 88%

Diffusion Model Alignment Using Direct Preference Optimization

Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, Nikhil Naik

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

2310.12773 2023-10-20 cs.AI cs.LG 88%

Safe RLHF: Safe Reinforcement Learning from Human Feedback

Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, Yaodong Yang

专题命中后训练与偏好优化：RLHF(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

2309.16240 2023-09-29 cs.LG cs.AI stat.ML 88%

Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints

Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, Yuxin Chen

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments Preprint

2308.08998 2023-08-22 cs.CL cs.LG 88%

Reinforced Self-Training (ReST) for Language Modeling

Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, Nando de Freitas

专题命中后训练与偏好优化：language model(title,abstract)；LLM(abstract)；large language model(abstract)；RLHF(abstract)

Comments 23 pages, 16 figures

2506.02018 2026-06-03 cs.CL 88%

Enhancing Paraphrase Type Generation: The Impact of DPO and RLHF Evaluated with Human-Ranked Data

增强释义类型生成：基于人工排序数据的DPO和RLHF评估影响

Christopher Lee Lübbers

机构 * University of Göttingen（哥廷根大学）

专题命中后训练与偏好优化：RLHF(title,title_cn)；language model(abstract)；preference optimization(abstract)；分类 cs.CL

AI总结本研究利用人工排序的释义类型数据集，结合直接偏好优化（DPO）使模型输出与人类判断对齐，将释义类型生成准确率提升3个百分点，人类偏好评分提升7个百分点，并创建了新的标注数据集以支持更严格的评估。

Comments 21 pages, 11 figures. Master's thesis, University of Goettingen, December 2024. Code: https://github.com/cluebbers/dpo-rlhf-paraphrase-types. Models: https://huggingface.co/collections/cluebbers/enhancing-paraphrase-type-generation-673ca8d75dfe2ce962a48ac0

2410.05451 2025-07-04 cs.CR cs.LG 88%

SecAlign: Defending Against Prompt Injection with Preference Optimization

Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, David Wagner, Chuan Guo

机构 * UC Berkeley / Meta（伯克利大学/Meta）； Meta

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract,comments)；large language model(abstract)；language model(abstract)

Comments ACM CCS 2025. Key words: prompt injection defense, LLM security, LLM-integrated applications

2412.00967 2024-12-03 cs.AI 88%

Linear Probe Penalties Reduce LLM Sycophancy

Henry Papadatos, Rachel Freedman

专题命中后训练与偏好优化：LLM(title,abstract)；language model(abstract,comments)；large language model(abstract)；RLHF(abstract)

Comments 20 pages, 15 figures, NeurIPS 2024 Workshop Socially Responsible Language Modelling Research (SoLaR)

2304.05302 2023-10-10 cs.CL 88%

RRHF: Rank Responses to Align Language Models with Human Feedback without tears

Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, Fei Huang

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；SFT(abstract)；RLHF(abstract)

Comments ArXiv version For NeurIPS 2023 accepted paper: RRHF: Rank Responses to Align Language Models with Human Feedback

1. 后训练与偏好优化 4541 篇

Eeyore: Realistic Depression Simulation via Supervised and Preference Optimization

SDPO: Segment-Level Direct Preference Optimization for Social Agents

CuDIP: Enhancing Theorem Proving in LLMs via Curriculum Learning-based Direct Preference Optimization

RoseRAG: Robust Retrieval-augmented Generation with Small-scale LLMs via Margin-aware Preference Optimization

Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment

Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment

Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment

Length Desensitization in Direct Preference Optimization

Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learning

$i$REPO: $i$mplicit Reward Pairwise Difference based Empirical Preference Optimization

Learning from Response not Preference: A Stackelberg Approach for LLM Detoxification using Non-parallel Data

Uncertainty-Penalized Direct Preference Optimization

Measuring memorization in RLHF for code completion

On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization

Disentangling Length from Quality in Direct Preference Optimization

Aligning Model Evaluations with Human Preferences: Mitigating Token Count Bias in Language Model Assessments

Unintended Impacts of LLM Alignment on Global Representation

Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement

Direct Alignment of Language Models via Quality-Aware Self-Refinement

LLM-FP4: 4-Bit Floating-Point Quantized Transformers

Improving Socratic Question Generation using Data Augmentation and Preference Optimization

ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference

Diffusion Model Alignment Using Direct Preference Optimization

Safe RLHF: Safe Reinforcement Learning from Human Feedback

Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints

Reinforced Self-Training (ReST) for Language Modeling

Enhancing Paraphrase Type Generation: The Impact of DPO and RLHF Evaluated with Human-Ranked Data

SecAlign: Defending Against Prompt Injection with Preference Optimization

Linear Probe Penalties Reduce LLM Sycophancy

RRHF: Rank Responses to Align Language Models with Human Feedback without tears