arXiv 2406.116572024-06-17cs.CLcs.CY

LLM 能否成为个性化评判者？

Can LLM be a Personalized Judge?

University of Cambridge（剑桥大学）

Yijiang River Dong, Tiancheng Hu, Nigel Collier

备注：: Our code is available at https://github.com/dong-river/Personalized-Judge

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文评估 LLM 基于人物画像充当个性化偏好评判者的可靠性，发现直接使用一致性不足，引入言语不确定性估计后，高确定性样本上的一致性超过 80%，并可媲美甚至超过第三方人工评估。

摘要：

随着大型语言模型（LLM）的用户群体在全球范围内扩展，确保其反映多样化的用户价值观和偏好至关重要。因此，研究界对 LLM 个性化日益增长的关注令人鼓舞。然而，当前工作通常依赖 LLM-as-a-Judge 方法进行评估，却没有彻底检验其有效性。本文研究 LLM-as-a-Personalized-Judge 的可靠性，即要求 LLM 基于人物画像判断用户偏好。研究结果表明，直接应用 LLM-as-a-Personalized-Judge 并不像此前假设的那样可靠，其与人类真实标注的一致性较低且不稳定。通常使用的人物画像往往过于简单，导致预测能力较低。为解决这些问题，我们将言语不确定性估计引入 LLM-as-a-Personalized-Judge 流程，使模型能够对不确定的判断表达低置信度。这一调整使二分类任务在高确定性样本上的一致性大幅提高（超过 80%）。通过人工评估，我们发现 LLM-as-a-Personalized-Judge 取得了与第三方人类评估相当的性能，并且在高确定性样本上甚至超过人类表现。本研究表明，增强确定性的 LLM-as-a-Personalized-Judge 为开发更可靠、可扩展的 LLM 个性化评估方法提供了有前景的方向。

英文摘要：

Ensuring that large language models (LLMs) reflect diverse user values and preferences is crucial as their user bases expand globally. It is therefore encouraging to see the growing interest in LLM personalization within the research community. However, current works often rely on the LLM-as-a-Judge approach for evaluation without thoroughly examining its validity. In this paper, we investigate the reliability of LLM-as-a-Personalized-Judge, asking LLMs to judge user preferences based on personas. Our findings suggest that directly applying LLM-as-a-Personalized-Judge is less reliable than previously assumed, showing low and inconsistent agreement with human ground truth. The personas typically used are often overly simplistic, resulting in low predictive power. To address these issues, we introduce verbal uncertainty estimation into the LLM-as-a-Personalized-Judge pipeline, allowing the model to express low confidence on uncertain judgments. This adjustment leads to much higher agreement (above 80%) on high-certainty samples for binary tasks. Through human evaluation, we find that the LLM-as-a-Personalized-Judge achieves comparable performance to third-party humans evaluation and even surpasses human performance on high-certainty samples. Our work indicates that certainty-enhanced LLM-as-a-Personalized-Judge offers a promising direction for developing more reliable and scalable methods for evaluating LLM personalization.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。