arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.04409cs.CLcs.AIcs.HC
收藏

解析人类对大语言模型的偏好:基于HUMAINE框架的民主意识评估

Unpacking Human Preference for LLMs: Demographically Aware Evaluation with the HUMAINE Framework

Nora Petrova, Andrew Gordon, Enzo Blindow

AI总结:

HUMAINE框架通过多维、人口统计学意识的评估方法,揭示了人类对大语言模型的偏好差异,发现年龄是主要分歧轴,且不同评估维度的判别能力差异显著。

摘要:

大型语言模型的评估面临重大挑战。技术基准测试往往缺乏现实世界的相关性,而现有的人类偏好评估则面临样本不具有代表性、评估深度浅显以及单一指标还原主义的问题。为了解决这些问题,我们引入HUMAINE框架,用于多维、具有人口统计学意识的测量人类与AI交互。我们收集了来自23,404名参与者的多轮自然对话,这些对话在22个人口统计学群体中进行分层,既包括美国也包括英国,以评估28种最先进的模型在五个以人类为中心的维度上的表现。我们使用一种分层贝叶斯布拉德利-蒂尔-戴维森(BTD)模型,并通过人口普查数据进行后分层,我们的分析揭示了三个关键发现。(1)我们建立了清晰的性能等级,其中google/gemini-2.5-pro在总体上排名第一,具有95.6%的后验概率为排名第一的模型。(2)我们发现显著的偏好异质性,用户年龄成为主要的人口统计学分歧轴;一个模型的感知排名可以在不同年龄组之间大幅变化,暴露了不具有代表性的样本通常掩盖的泛化失败。(3)我们量化了在评估维度之间判别能力的巨大差异,模糊的质量如信任、伦理与安全显示出65%的平票率,与总体冠军10%的决定性平票率形成鲜明对比。我们的工作强调了在LLM评估中需要更多维、具有人口统计学意识的视角。我们发布了完整数据集、互动排行榜和开源框架。

英文摘要:

The evaluation of large language models faces significant challenges. Technical benchmarks often lack real-world relevance, while existing human preference evaluations suffer from unrepresentative sampling, superficial assessment depth, and single-metric reductionism. To address these issues, we introduce HUMAINE, a framework for multidimensional, demographically aware measurement of human-AI interaction. We collected multi-turn, naturalistic conversations from 23,404 participants that were stratified across 22 demographic groups, both in the US and UK, to evaluate 28 state-of-the-art models across five human-centric dimensions. We use a hierarchical Bayesian Bradley-Terry-Davidson (BTD) model, with post-stratification to census data, and our analysis reveals three key insights. \textbf{(1)} We establish a clear performance hierarchy where \texttt{google/gemini-2.5-pro} ranks first overall, with a 95.6\% posterior probability of being the top-ranked model. \textbf{(2)} We uncover significant preference heterogeneity, with user age emerging as the primary demographic axis of disagreement; a model's perceived rank can shift substantially across age groups, exposing failures in generalisation that unrepresentative samples typically mask. \textbf{(3)} We quantify the vast difference in discriminative power across evaluation dimensions, with ambiguous qualities like \textit{Trust, Ethics \& Safety} showing a 65\% tie rate, in stark contrast to the decisive 10\% tie rate for \textit{Overall Winner}. Our work emphasises the need for a more multidimensional, demographically aware perspective in LLM evaluation. We release our complete dataset, interactive leaderboard, and open-source framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑