HAL: 通过对齐引导LLM的人类相似性
HAL: Inducing Human-likeness in LLMs with Alignment
AI总结：
提出HAL框架，利用可解释的数据驱动奖励信号，通过对比对话数据提取显式会话特征并优化偏好，使模型在对话中更常被视为类人，同时保持整体性能。
摘要：
将语言模型与定性行为特征（如人类相似性）对齐仍然困难，因为这些特征难以定义、测量和优化。因此，类人行为的改进主要由规模或广泛的监督训练驱动，而非有针对性的对齐。我们引入人类对齐LLMs（HAL），这是一个使用可解释、数据驱动奖励将语言模型与对话人类相似性对齐的框架。HAL从对比对话数据中推导出显式会话特征，将其组合成紧凑的标量分数，并作为透明奖励信号用于标准偏好优化方法的对齐。使用这种方法，我们对不同大小的模型进行对齐，而不影响其整体性能。在大规模Chatbot Arena风格的人类评估中，与HAL对齐的模型在对话中更常被视为类人。由于HAL基于显式、可解释的特征，它能够检查对齐行为并诊断意外效果。更广泛地说，HAL展示了语言的软定性属性——以前超出对齐范围——如何以可解释和可解释的方式变得可测量和可对齐。
英文摘要：
Aligning language models to qualitative behavioral traits, such as human-likeness, remains difficult because they are hard to define, measure, and optimize. As a result, improvements in human-like behavior are largely driven by scale or broad supervised training, rather than targeted alignment. We introduce Human Aligning LLMs (HAL), a framework for aligning language models to conversational human-likeness using an interpretable, data-driven reward. HAL derives explicit conversational traits from contrastive dialogue data, combines them into a compact scalar score, and uses this score as a transparent reward signal for alignment with standard preference optimization methods. Using this approach, we align models of varying sizes without affecting their overall performance. In large-scale Chatbot Arena-style human evaluations, a model aligned with HAL is more frequently perceived as human-like in conversation. Because HAL operates over explicit, interpretable traits, it enables inspection of alignment behavior and diagnosis of unintended effects. More broadly, HAL demonstrates how soft, qualitative properties of language--previously outside the scope for alignment--can be made measurable and aligned in an interpretable and explainable way.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
