揭示因子层面偏好以提升人机对齐水平
Uncovering Factor Level Preferences to Improve Human-Model Alignment
AI总结：
针对现有大语言模型对齐评估粗粒度、缺乏可解释性的问题，提出PROFILE框架开展因子层面偏好对齐分析，发现模型生成与判别任务的对齐差距，可用于提升人机对齐效果。
摘要：
大语言模型（LLM）常表现出与人类偏好相悖的倾向，例如偏好特定写作风格或生成过于冗长的输出。尽管识别导致这类错位的因素对模型优化至关重要，但现有评估方法依赖粗粒度比较且缺乏可解释性，使得该识别工作仍颇具挑战。为解决这一问题，我们提出PROFILE自动化框架，用于揭示并衡量人类与大语言模型在因子层面的偏好对齐情况。借助PROFILE，我们分析了摘要生成、指令遵循、基于文档的问答三类核心任务中的偏好对齐表现，发现了显著差异：大语言模型生成文本时与人类偏好的因子层面对齐度较差，但在判别任务中表现出较强的对齐性。我们验证了利用所发现的生成-判别差距，可通过包括自引导微调在内的多种方法提升大语言模型的对齐效果。本研究凸显了因子层面分析在识别隐式错位问题上的价值，也为提升大语言模型与人类偏好的对齐水平提供了实用框架。
英文摘要：
Large language models (LLMs) often exhibit tendencies that diverge from human preferences, such as favoring certain writing styles or producing overly verbose outputs. While crucial for improvement, identifying the factors driving these misalignments remains challenging due to existing evaluation methods' reliance on coarse-grained comparisons and lack of explainability. To address this, we introduce PROFILE, an automated framework to uncover and measure factor-level preference alignment of humans and LLMs. Using PROFILE, we analyze preference alignment across three key tasks: summarization, instruction-following, and document-based QA. We find a significant discrepancy: while LLMs show poor factor-level alignment with human preferences when generating texts, they demonstrate strong alignment in discrimination tasks. We demonstrate how leveraging the identified generation-discrimination gap can be used to improve LLM alignment through multiple approaches, including fine-tuning with self-guidance. Our work highlights the value of factor-level analysis for identifying hidden misalignments and provides a practical framework for improving LLM-human preference alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
