带个性与自信的判断：关于基于个性的LLM相关性评估的研究
Judging with Personality and Confidence: A Study on Personality-Conditioned LLM Relevance Assessment
AI总结：
本研究探讨了基于人格的LLM相关性评估，发现不同人格特质影响置信度和相关性判断，通过随机森林分类器提升评估性能。
摘要：
最近的研究表明，提示可以使大型语言模型（LLMs）模拟特定的人格特质并产生与这些特质一致的行为。然而，对这些模拟的人格如何影响关键的网络搜索决策，特别是相关性评估，仍缺乏深入理解。此外，很少有研究探讨模拟的人格如何影响自信校准，特别是倾向于过度自信或过度自信的倾向。尽管心理学文献表明这些偏见是特质特定的，通常将高外向性与过度自信、高神经质与过度自信联系起来，这一空白仍然存在。为了解决这一空白，我们进行了全面研究，评估了多个LLM，包括商业模型和开源模型，并提示它们模拟大五人格特质。我们测试了这些模型在三个测试集（TREC DL 2019、TREC DL 2020和LLMJudge）上的表现，收集了每个查询-文档对的两个关键输出：相关性判断和自我报告的置信度分数。研究结果表明，如低宜人性的人格在人类标签上的一致性比未提示条件更紧密。此外，低尽责性在抑制过度自信和过度自信方面表现良好。我们还观察到，相关性评分和置信度分布在不同人格之间系统性地变化。基于以上发现，我们将基于人格的评分和置信度作为特征纳入随机森林分类器中。这种方法在新数据集（TREC DL 2021）上实现了超越最佳单人格条件的性能，即使在有限的训练数据下也是如此。这些发现表明，基于人格的置信度提供了一个互补的预测信号，为更可靠和与人类一致的LLM评估器铺平了道路。
英文摘要：
Recent studies have shown that prompting can enable large language models (LLMs) to simulate specific personality traits and produce behaviors that align with those traits. However, there is limited understanding of how these simulated personalities influence critical web search decisions, specifically relevance assessment. Moreover, few studies have examined how simulated personalities impact confidence calibration, specifically the tendencies toward overconfidence or underconfidence. This gap exists even though psychological literature suggests these biases are trait-specific, often linking high extraversion to overconfidence and high neuroticism to underconfidence. To address this gap, we conducted a comprehensive study evaluating multiple LLMs, including commercial models and open-source models, prompted to simulate Big Five personality traits. We tested these models across three test collections (TREC DL 2019, TREC DL 2020, and LLMJudge), collecting two key outputs for each query-document pair: a relevance judgment and a self-reported confidence score. The findings show that personalities such as low agreeableness consistently align more closely with human labels than the unprompted condition. Additionally, low conscientiousness performs well in balancing the suppression of both overconfidence and underconfidence. We also observe that relevance scores and confidence distributions vary systematically across different personalities. Based on the above findings, we incorporate personality-conditioned scores and confidence as features in a random forest classifier. This approach achieves performance that surpasses the best single-personality condition on a new dataset (TREC DL 2021), even with limited training data. These findings highlight that personality-derived confidence offers a complementary predictive signal, paving the way for more reliable and human-aligned LLM evaluators.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
