语言大模型 / LLM - arXivDaily 专题

2507.20133 2025-07-30 cs.CL cs.AI cs.LG 85%

Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering

Anas Mohamed, Azal Ahmad Khan, Xinran Wang, Ahmad Faraz Khan, Shuwen Ge, Saman Bahzad Khan, Ayaan Ahmad, Ali Anwar

机构 * University of Minnesota（明尼苏达大学）； Virginia Tech（弗吉尼亚理工大学）； Xi’an University of Technology（西安理工大学）； Lahore University of Management Sciences（拉合尔管理科学大学）； University of California, Santa Cruz（加州大学圣克鲁兹分校）

专题命中后训练与偏好优化：preference optimization(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2506.14903 2025-06-19 cs.CV 85%

DETONATE: A Benchmark for Text-to-Image Alignment and Kernelized Direct Preference Optimization

Renjith Prasad, Abhilekh Borah, Hasnat Md Abdullah, Chathurangi Shyalika, Gurpreet Singh, Ritvik Garimella, Rajarshi Roy, Harshul Surana, Nasrin Imanpour, Suranjana Trivedy, Amit Sheth, Amitava Das

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

Comments 59 pages, 10 figures

2410.15334 2025-06-09 cs.CV 85%

Modality-Fair Preference Optimization for Trustworthy MLLM Alignment

Songtao Jiang, Yan Zhang, Ruizhe Chen, Tianxiang Hu, Yeying Jin, Qinglin He, Yang Feng, Jian Wu, Zuozhu Liu

机构 * Zhejiang University（浙江大学）； ByteDance（字节跳动）； National University of Singapore（新加坡国立大学）； Angelalign Inc., China（中国Angelalign公司）

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

2505.22396 2025-05-29 cs.CV 85%

Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs

Xudong Li, Mengdan Zhang, Peixian Chen, Xiawu Zheng, Yan Zhang, Jingyuan Zheng, Yunhang Shen, Ke Li, Chaoyou Fu, Xing Sun, Rongrong Ji

机构 * Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）； Tencent Youtu Lab（腾讯优图实验室）； Nanjing University（南京大学）

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

2402.10184 2025-05-29 cs.LG cs.AI cs.CL cs.DM 85%

Reward Generalization in RLHF: A Topological Perspective

Tianyi Qiu, Fanzhi Zeng, Jiaming Ji, Dong Yan, Kaile Wang, Jiayi Zhou, Yang Han, Josef Dai, Xuehai Pan, Yaodong Yang

机构 * Center for AI Safety and Governance, Institute for AI, Peking University（人工智能安全与治理中心，人工智能研究院，北京大学）； Tsinghua University（清华大学）； Baichuan Inc（北川公司）

专题命中后训练与偏好优化：RLHF(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 46 pages, ACL 2025 (Findings)

2505.20627 2025-05-28 cs.GT stat.ML 85%

Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching

Zhekun Shi, Kaizhao Liu, Qi Long, Weijie J. Su, Jiancong Xiao

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)

2505.19307 2025-05-27 cs.IR 85%

Aligning Web Query Generation with Ranking Objectives via Direct Preference Optimization

João Coelho, Bruno Martins, João Magalhães, Chenyan Xiong

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

Comments Accepted at SIGIR 2025

2505.01706 2025-05-06 cs.AI cs.CL cs.LG 85%

Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm

Sarvesh Shashidhar, Ritik, Nachiketa Patil, Suraj Racha, Ganesh Ramakrishnan

机构 * Centre for Machine Intelligence and Data Science, IIT Bombay（印度理工学院班加罗尔机器智能与数据科学中心）； Koita Centre for Digital Health, IIT Bombay（印度理工学院班加罗尔数字健康中心）； Department of Computer Science and Engineering, IIT Bombay（印度理工学院班加罗尔计算机科学与工程系）

专题命中后训练与偏好优化：preference optimization(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Updated abstract, algorithm and experimental results

2410.08847 2025-04-29 cs.LG cs.AI cs.CL stat.ML 85%

Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization

Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, Boris Hanin

机构 * Princeton Language and Intelligence, Princeton University（普林斯顿语言与智能，普林斯顿大学）

专题命中后训练与偏好优化：preference optimization(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to ICLR 2025; Code available at https://github.com/princeton-nlp/unintentional-unalignment

2504.15176 2025-04-22 cs.CV 85%

DSPO: Direct Semantic Preference Optimization for Real-World Image Super-Resolution

Miaomiao Cai, Simiao Li, Wei Li, Xudong Huang, Hanting Chen, Jie Hu, Yunhe Wang

机构 * University of Science and Technology of China（中国科学技术大学）； Huawei Noah’s Ark Lab（华为诺亚实验室）

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

2503.10965 2025-03-31 cs.AI cs.CL cs.LG 85%

Auditing language models for hidden objectives

Samuel Marks, Johannes Treutlein, Trenton Bricken, Jack Lindsey, Jonathan Marcus, Siddharth Mishra-Sharma, Daniel Ziegler, Emmanuel Ameisen, Joshua Batson, Tim Belonax, Samuel R. Bowman, Shan Carter, Brian Chen, Hoagy Cunningham, Carson Denison, Florian Dietz, Satvik Golechha, Akbir Khan, Jan Kirchner, Jan Leike, Austin Meek, Kei Nishimura-Gasparian, Euan Ong, Christopher Olah, Adam Pearce, Fabien Roger, Jeanne Salle, Andy Shih, Meg Tong, Drake Thomas, Kelley Rivoire, Adam Jermyn, Monte MacDiarmid, Tom Henighan, Evan Hubinger

专题命中后训练与偏好优化：language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.11455 2025-02-18 cs.CR 85%

Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training

Fenghua Weng, Jian Lou, Jun Feng, Minlie Huang, Wenjie Wang

专题命中后训练与偏好优化：language model(title,abstract)；LLM(abstract)；large language model(abstract)

2501.06761 2025-01-14 cs.CV 85%

VidChain: Chain-of-Tasks with Metric-based Direct Preference Optimization for Dense Video Captioning

Ji Soo Lee, Jongha Kim, Jeehye Na, Jinyoung Park, Hyunwoo J. Kim

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

Comments AAAI 2025

2409.06691 2024-12-31 cs.LG cs.AI cs.CL 85%

Geometric-Averaged Preference Optimization for Soft Preference Labels

Hiroki Furuta, Kuang-Huei Lee, Shixiang Shane Gu, Yutaka Matsuo, Aleksandra Faust, Heiga Zen, Izzeddin Gur

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at NeurIPS 2024

2402.08925 2024-12-30 cs.CL cs.AI cs.LG cs.RO 85%

MaxMin-RLHF: Alignment with Diverse Human Preferences

Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, Mengdi Wang

专题命中后训练与偏好优化：RLHF(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2411.11909 2024-11-26 cs.CV 85%

SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization

Hongrui Jia, Chaoya Jiang, Haiyang Xu, Wei Ye, Mengfan Dong, Ming Yan, Ji Zhang, Fei Huang, Shikun Zhang

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

2410.17126 2024-10-23 cs.CL cs.AI cs.LG 85%

Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards

Alexander G. Padula, Dennis J. N. J. Soemers

专题命中后训练与偏好优化：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at BNAIC 2024

2409.09603 2024-09-17 cs.AI cs.CL cs.LG 85%

Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison

Judy Hanwen Shen, Archit Sharma, Jun Qin

专题命中后训练与偏好优化：RLHF(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Working Paper

2404.05530 2024-08-07 cs.CL cs.AI cs.CR cs.LG 85%

Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data

Tim Baumgärtner, Yang Gao, Dana Alon, Donald Metzler

专题命中后训练与偏好优化：RLHF(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2310.01377 2024-07-17 cs.CL cs.AI cs.LG 85%

UltraFeedback: Boosting Language Models with Scaled AI Feedback

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, Maosong Sun

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICML 2024 camera ready

2406.07971 2024-06-14 cs.CL cs.AI cs.LG 85%

It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF

Taiming Lu, Lingfeng Shen, Xinyu Yang, Weiting Tan, Beidi Chen, Huaxiu Yao

专题命中后训练与偏好优化：RLHF(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2402.02992 2024-05-27 cs.LG cs.AI cs.CL 85%

Decoding-time Realignment of Language Models

Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, Mathieu Blondel

专题命中后训练与偏好优化：language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments In Proceedings of the 41st International Conference on Machine Learning (ICML 2024)

2404.08495 2024-04-17 cs.LG cs.AI cs.CL 85%

Dataset Reset Policy Optimization for RLHF

Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D. Lee, Wen Sun

专题命中后训练与偏好优化：RLHF(title,abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 28 pages, 6 tables, 3 Figures, 3 Algorithms

2311.16714 2024-04-01 cs.CV 85%

Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld

Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lusong Li, Li Shen, Xiaodong He, Jing Jiang, Yuhui Shi

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)

Comments Accepted by CVPR 2024

2303.16755 2024-02-26 cs.CL cs.AI cs.LG 85%

Training Language Models with Language Feedback at Scale

Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, Ethan Perez

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Published in TMLR: https://openreview.net/forum?id=xo3hI5MwvU

2402.11882 2024-02-20 cs.CV 85%

NOTE: Notable generation Of patient Text summaries through Efficient approach based on direct preference optimization

Imjin Ahn, Hansle Gwon, Young-Hak Kim, Tae Joon Jun, Sanghyun Park

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

Comments 13 pages, 3 figures, 5 tables

2402.07319 2024-02-13 cs.LG cs.AI cs.CL 85%

ODIN: Disentangled Reward Mitigates Hacking in RLHF

Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, Bryan Catanzaro

专题命中后训练与偏好优化：RLHF(title,abstract)；LLM(abstract)；分类 cs.CL、cs.AI、cs.LG

2401.16335 2024-01-30 cs.LG cs.AI cs.CL stat.ML 85%

Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF

Banghua Zhu, Michael I. Jordan, Jiantao Jiao

专题命中后训练与偏好优化：RLHF(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2310.11589 2023-10-19 cs.CL cs.AI cs.LG 85%

Eliciting Human Preferences with Language Models

Belinda Z. Li, Alex Tamkin, Noah Goodman, Jacob Andreas

专题命中后训练与偏好优化：language model(title,abstract)；prompting(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 26 pages, 15 figures

2212.09251 2022-12-20 cs.CL cs.AI cs.LG 85%

Discovering Language Model Behaviors with Model-Written Evaluations

Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan

专题命中后训练与偏好优化：language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments for associated data visualizations, see https://www.evals.anthropic.com/model-written/ for full datasets, see https://github.com/anthropics/evals

1. 后训练与偏好优化 4556 篇

Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering

DETONATE: A Benchmark for Text-to-Image Alignment and Kernelized Direct Preference Optimization

Modality-Fair Preference Optimization for Trustworthy MLLM Alignment

Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs

Reward Generalization in RLHF: A Topological Perspective

Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching

Aligning Web Query Generation with Ranking Objectives via Direct Preference Optimization

Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm

Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization

DSPO: Direct Semantic Preference Optimization for Real-World Image Super-Resolution

Auditing language models for hidden objectives

Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training

VidChain: Chain-of-Tasks with Metric-based Direct Preference Optimization for Dense Video Captioning

Geometric-Averaged Preference Optimization for Soft Preference Labels

MaxMin-RLHF: Alignment with Diverse Human Preferences

SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization

Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards

Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison

Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data

UltraFeedback: Boosting Language Models with Scaled AI Feedback

It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF

Decoding-time Realignment of Language Models

Dataset Reset Policy Optimization for RLHF

Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld

Training Language Models with Language Feedback at Scale

NOTE: Notable generation Of patient Text summaries through Efficient approach based on direct preference optimization

ODIN: Disentangled Reward Mitigates Hacking in RLHF

Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF

Eliciting Human Preferences with Language Models

Discovering Language Model Behaviors with Model-Written Evaluations