大语言模型中的评论赞誉导向：来自电影偏好诱导的证据
Critical Acclaim Orientation in Large Language Models: Evidence from Film Preference Elicitation
AI总结：
本研究以8种大语言模型为对象，通过200部电影的基准测试，发现模型普遍呈现评论赞誉导向，该导向随模型规模增强，且提示框架会影响评价排名。
AI中文摘要：
大语言模型（LLM）在包含人类对电影、书籍、音乐等内容评价的语料库上训练，但LLM是否会系统性地复制评价等级尚不清楚。此前关于LLM文化偏见的研究提出了相互矛盾的预期：模型可能会镜像网络文本的流行信号，或复制评论话语中嵌入的声望形式。本研究针对该问题，采用四个系列（Anthropic、OpenAI、Alibaba、Mistral）的8种模型，以包含200部电影的基准数据集（分为评论赞誉型、商业成功型、双重合法性型：评论赞誉+商业成功）开展电影评价研究。对每个模型开展20000次成对强制选择比较并使用Bradley-Terry估计分析后，所有模型均呈现一致的评论赞誉导向：相较于商业成功但未获评论认可的电影，模型会选择评论赞誉但商业表现平淡的电影，且该模式在每个系列内随模型规模增大而增强。此外，嵌套OLS回归分析显示，评价导向、公众可见度与大众接受度可分别解释偏好；调整公众可见度后，模型对双重合法性电影的偏好会反转，转向仅评论赞誉的电影，同时纳入大众接受度后，仅商业成功电影的劣势会大幅减弱。最后，评价导向与推荐导向的提示框架会产生不同的排名，表明评论赞誉导向可能在LLM的实际部署中间接显现。
英文摘要：
Large language models (LLMs) are trained on corpora that contain expressions of human judgment about films, books, music, and more. Yet whether LLMs systematically reproduce evaluative hierarchies remains unclear. Prior research on cultural bias in LLMs suggests competing expectations: models may mirror the popularity signals of internet texts, or may reproduce forms of prestige embedded in critical discourse. We probe this question through a study of film evaluations with eight models from four families (Anthropic, OpenAI, Alibaba, and Mistral), using a 200-film benchmark partitioned into critically acclaimed, commercially successful, and dual-legitimacy (critical acclaim + commercial success) films. Across 20,000 pairwise forced-choice comparisons per model analyzed with Bradley--Terry estimation, we observe a consistent critical acclaim orientation with all models: critically acclaimed yet commercially obscure films are selected over commercially successful yet critically unrecognized ones. This pattern grows with model scale within each family. In addition, nested OLS regression analyses show that evaluative orientation, public visibility, and popular reception distinctly help explain preferences. Adjusting for public visibility reverses the models' preference for dual-legitimacy films over critical acclaim-only films, while additionally accounting for popular reception attenuates much of the disadvantage of films with commercial success only. Finally, evaluative and recommendation-oriented prompt framings produce divergent rankings, suggesting that critical acclaim orientation may manifest indirectly in real-world LLM deployments.
