LLMs能否捕捉专家不确定性？对人类学质性研究中价值对齐的比较分析
Can LLMs Capture Expert Uncertainty? A Comparative Analysis of Value Alignment in Ethnographic Qualitative Research
AI总结：
本文研究LLMs在人类学质性研究中捕捉专家不确定性的能力，发现Qwen在价值对齐上表现最佳，但需进一步探讨模型偏差问题。
摘要：
开放性访谈的质性分析在人类学和经济学研究中扮演着核心角色，通过揭示个体的价值观、动机以及文化嵌入的金融行为。尽管大型语言模型（LLMs）为自动化和丰富此类解释性工作提供了有希望的支持，但其在存在内在任务模糊性的情况下产生细致、可靠的解释能力仍不明确。在我们的工作中，我们评估了LLMs在识别长篇访谈中表达的前三个人类价值观的任务上的表现，基于Schwartz基本价值观理论框架。我们将其输出与专家注释进行比较，分析其在性能和不确定性模式上与专家的相对表现。结果表明，LLMs在集合度量（F1、Jaccard）上接近人类水平，但在恢复精确价值观排名方面存在困难，这体现在较低的RBO分数上。虽然大多数模型的Schwartz价值观分布平均与人类分析师接近，但其在Schwartz价值观上的不确定性结构却与专家的不确定性模式不同。在评估的模型中，Qwen最接近专家级别的同意，并且在Schwartz价值观分布上表现出最强的对齐性。LLM集成方法在各种指标上都产生了持续的提升，其中多数投票和波尔多计数表现最佳。值得注意的是，对某些Schwartz价值观的系统性强调，如安全，表明LLMs不仅有可能提供互补的视角，也表明需要进一步研究模型引起的价值观偏差。总体而言，我们的发现突显了LLMs作为在本质上模糊的质性价值分析中的合作者的潜力和局限性。
英文摘要：
Qualitative analysis of open-ended interviews plays a central role in ethnographic and economic research by uncovering individuals' values, motivations, and culturally embedded financial behaviors. While large language models (LLMs) offer promising support for automating and enriching such interpretive work, their ability to produce nuanced, reliable interpretations under inherent task ambiguity remains unclear. In our work we evaluate LLMs on the task of identifying the top three human values expressed in long-form interviews based on the Schwartz Theory of Basic Values framework. We compare their outputs to expert annotations, analyzing both performance and uncertainty patterns relative to the experts. Results show that LLMs approach the human ceiling on set-based metrics (F1, Jaccard) but struggle to recover exact value rankings, as reflected in lower RBO scores. While the average Schwartz value distributions of most models closely match those of human analysts, their uncertainty structures across the Schwartz values diverge from expert uncertainty patterns. Among the evaluated models, Qwen performs closest to expert-level agreement and exhibits the strongest alignment with expert Schwartz value distributions. LLM ensemble methods yield consistent gains across metrics, with Majority Vote and Borda Count performing best. Notably, systematic overemphasis on certain Schwartz values, like Security, suggests both the potential of LLMs to provide complementary perspectives and the need to further investigate model-induced value biases. Overall, our findings highlight both the promise and the limitations of LLMs as collaborators in inherently ambiguous qualitative value analysis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
