语言大模型 / LLM

InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization

Yunan Wang, Jijie Li, Bo-Wen Zhang, Liangdong Wang, Guang Liu

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

TLDR: Token-Level Detective Reward Model for Large Vision Language Models

Deqing Fu, Tong Xiao, Rui Wang, Wang Zhu, Pengchuan Zhang, Guan Pang, Robin Jia, Lawrence Chen

专题命中 后训练与偏好优化 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

Comments Published as a conference paper at ICLR 2025

Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF

Nuoya Xiong, Aarti Singh

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；language model(abstract)；分类 cs.AI、cs.LG

Process Reward Models for LLM Agents: Practical Framework and Directions

Sanjiban Choudhury

专题命中 后训练与偏好优化 ：LLM(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments 17 pages, 7 figures

Post-Training Statistical Calibration for Higher Activation Sparsity

Vui Seng Chua, Yujie Pan, Nilesh Jain

专题命中 后训练与偏好优化 ：post-training(title,abstract)；LLM(abstract)；分类 cs.AI、cs.LG

Comments ENLSP-IV NeurIPS Workshop 2024

SimPO: Simple Preference Optimization with a Reference-Free Reward

Yu Meng, Mengzhou Xia, Danqi Chen

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2024. Code & models: https://github.com/princeton-nlp/SimPO. v3 updates: Gemma 2 results (Appendix J); more discussions about length normalization (Section 2.2) and KL regularization (Section 2.3)

InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling

Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, Dacheng Tao

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments The paper has been accepted by NeurIPS 2024

Noise Contrastive Alignment of Language Models with Explicit Rewards

Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, Jun Zhu

专题命中 后训练与偏好优化 ：language model(title,abstract)；preference optimization(abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2024

Preference Alignment Improves Language Model-Based TTS

Jinchuan Tian, Chunlei Zhang, Jiatong Shi, Hao Zhang, Jianwei Yu, Shinji Watanabe, Dong Yu

专题命中 后训练与偏好优化 ：language model(title,abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI

Decoding Multilingual Moral Preferences: Unveiling LLM's Biases Through the Moral Machine Experiment

Karina Vida, Fabian Damken, Anne Lauscher

专题命中 后训练与偏好优化 ：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments to be published in AIES 2024 Proceedings

Direct Preference Optimization for Suppressing Hallucinated Prior Exams in Radiology Report Generation

Oishi Banerjee, Hong-Yu Zhou, Subathra Adithan, Stephen Kwak, Kay Wu, Pranav Rajpurkar

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Added acknowledgemnts

RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback

Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, Zackory Erickson

专题命中 后训练与偏好优化 ：foundation model(title,abstract)；prompting(abstract)；分类 cs.AI、cs.LG

Comments ICML 2024

Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimization

Yi Gu, Zhendong Wang, Yueqin Yin, Yujia Xie, Mingyuan Zhou

专题命中 后训练与偏好优化 ：preference optimization(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Generalized Preference Optimization: A Unified Approach to Offline Alignment

Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, Bilal Piot

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments Accepted at ICML 2023 main conference

RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation

Chanwoo Park, Mingyang Liu, Dingwen Kong, Kaiqing Zhang, Asuman Ozdaglar

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments Added experiments

Leveraging Domain Knowledge for Efficient Reward Modelling in RLHF: A Case-Study in E-Commerce Opinion Summarization

Swaroop Nath, Tejpalsingh Siledar, Sankara Sri Raghava Ravindra Muddu, Rupasai Rangaraju, Harshad Khadilkar, Pushpak Bhattacharyya, Suman Banerjee, Amey Patil, Sudhanshu Shekhar Singh, Muthusamy Chelliah, Nikesh Garera

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments 19 pages, 6 figures, 21 tables

Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF

Anand Siththaranjan, Cassidy Laidlaw, Dylan Hadfield-Menell

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；LLM(abstract)；分类 cs.AI、cs.LG

Comments Presented at ICLR 2024

Mixture-of-Depths: Dynamically allocating compute in transformer-based language models

David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, Adam Santoro

专题命中 后训练与偏好优化 ：language model(title,abstract)；post-training(abstract)；分类 cs.CL、cs.LG

Underspecification in Language Modeling Tasks: A Causality-Informed Study of Gendered Pronoun Resolution

Emily McMilin

专题命中 后训练与偏好优化 ：language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments 24 pages, 41 figures

Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation

Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, Lei Meng

专题命中 后训练与偏好优化 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Chain of Hindsight Aligns Language Models with Feedback

Hao Liu, Carmelo Sferrazza, Pieter Abbeel

专题命中 后训练与偏好优化 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

Confronting Reward Model Overoptimization with Constrained RLHF

Ted Moskovitz, Aaditya K. Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D. Dragan, Stephen McAleer

专题命中 后训练与偏好优化 ：RLHF(title)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Aligning Language Models with Preferences through f-divergence Minimization

Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, Marc Dymetman

专题命中 后训练与偏好优化 ：language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, Jack Clark

专题命中 后训练与偏好优化 ：language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Teaching language models to support answers with verified quotes

Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, Nat McAleese

专题命中 后训练与偏好优化 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

An Effective Domain Adaptive Post-Training Method for BERT in Response Selection

Taesun Whang, Dongyub Lee, Chanhee Lee, Kisu Yang, Dongsuk Oh, HeuiSeok Lim

专题命中 后训练与偏好优化 ：post-training(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments INTERSPEECH 2020

Parameter Exploration for RLVR via Variational Learning

基于变分学习的RLVR参数探索

Vatsal Venkatkrishna, Nico Daheim, Iryna Gurevych

机构 * INSAIT, Sofia University “St. Kliment Ohridski”（INSAIT，圣克莱门特奥赫里德斯基索非亚大学） ； National Research Center for Applied Cybersecurity ATHENE（ATHENE国家应用网络安全研究中心） 纠错

专题命中 后训练与偏好优化 ：LLM(summary_cn,abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 本文针对LLM强化学习的探索问题，提出参数空间探索思路，引入3PO方法，在OLMo-3-1025-7B等模型的数学推理等任务上，以相近计算成本相比GRPO提升性能，减少训练中的异常分组与轨迹。

GraphQAG: A Knowledge-Graph-Guided Visual Analytics Framework for Question-Answer Pairs Generation

GraphQAG：一种用于问答对生成的知识图谱引导型可视分析框架

Yize Li, Ruiqi Yu, Tianya Pan, Ningxin Li, Songyue Li, Xiangyang Wu, Jinchang Li, Zhiguang Zhou

专题命中 后训练与偏好优化 ：LLM(abstract,abstract_cn)；large language model(abstract)；language model(abstract)；post-training(abstract)

AI总结 该研究针对长文档知识碎片化的问题，提出GraphQAG知识图谱引导型可视分析框架，经评估可帮助用户优化问答对集合，提升问答对的全面性与可信度。

DPO-F+: Aligning Code Repair Feedback with Developers' Preferences

DPO-F+：使代码修复反馈与开发者偏好对齐

Zihan Fang, Yifan Zhang, Yueke Zhang, Kevin Leach, Yu Huang

专题命中 后训练与偏好优化 ：LLM(abstract,abstract_cn)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

AI总结 研究针对大语言模型在代码修复中开发者难解读输出的问题，提出DPO-f+框架，通过定义指标、构建数据集、微调模型及评估反馈质量，提升反馈准确性与对齐度，增强了代码理解和人机协作。

Comments 10 pages, 2 figures

Query-Centric Optimization of AI Workflows via Approximate Query Processing and Proxy Models

基于近似查询处理和代理模型的AI工作流查询中心优化

Huayi Wang, Jun Xu, Gromit Yeuk-Yin Chan

专题命中 后训练与偏好优化 ：LLM(summary_cn,abstract)；post-training(abstract)

AI总结 提出将AI工作流视为声明式查询，利用近似查询处理（AQP）和代理模型（PM）过滤减少昂贵模型调用，在TPC-DS和LLM流水线上实现85-90%和60-70%的调用减少。