HLB：基准测试大语言模型在语言使用中的人类相似性
HLB: Benchmarking LLMs' Humanlikeness in Language Use
AI总结：
本文提出人类相似性基准（HLB），通过10项心理语言学实验和2000余名人类数据对比，系统评估20个大语言模型在语言使用中的人类相似性，发现模型其他性能提升未必增加人类相似性。
摘要：
随着合成数据在训练语言模型中日益普遍，特别是通过生成对话，人们开始担心这些模型可能偏离真实人类语言模式，可能失去人类交流中固有的丰富性与创造力。这凸显了评估语言模型在真实语言使用中人类相似性的迫切需求。本文提出了一个全面的人类相似性基准（HLB），使用10项心理语言学实验评估20个大语言模型（LLM），旨在探究核心语言学层面，包括语音、词汇、句法、语义和语篇（见 https://huggingface.co/spaces/XufengDuan/HumanLikeness）。为确立比较基准，我们收集了2000多名人类参与者的回应，并将其与LLM在这些实验中的输出进行比较。为进行严格评估，我们开发了一种编码算法，能准确识别语言使用模式，从而提取每项任务的回应分布。通过比较人类参与者与LLM的回应分布，我们基于分布相似性量化了人类相似性。结果显示，LLM在各个语言层面复现人类回应的能力存在细粒度差异。重要的是，我们发现其他性能指标的改进未必带来人类相似性的提升，在某些情况下甚至导致下降。通过将心理语言学方法引入模型评估，该基准提供了首个系统评估LLM语言使用人类相似性的框架。
英文摘要：
As synthetic data becomes increasingly prevalent in training language models, particularly through generated dialogue, concerns have emerged that these models may deviate from authentic human language patterns, potentially losing the richness and creativity inherent in human communication. This highlights the critical need to assess the humanlikeness of language models in real-world language use. In this paper, we present a comprehensive humanlikeness benchmark (HLB) evaluating 20 large language models (LLMs) using 10 psycholinguistic experiments designed to probe core linguistic aspects, including sound, word, syntax, semantics, and discourse (see https://huggingface.co/spaces/XufengDuan/HumanLikeness). To anchor these comparisons, we collected responses from over 2,000 human participants and compared them to outputs from the LLMs in these experiments. For rigorous evaluation, we developed a coding algorithm that accurately identified language use patterns, enabling the extraction of response distributions for each task. By comparing the response distributions between human participants and LLMs, we quantified humanlikeness through distributional similarity. Our results reveal fine-grained differences in how well LLMs replicate human responses across various linguistic levels. Importantly, we found that improvements in other performance metrics did not necessarily lead to greater humanlikeness, and in some cases, even resulted in a decline. By introducing psycholinguistic methods to model evaluation, this benchmark offers the first framework for systematically assessing the humanlikeness of LLMs in language use.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
