大语言模型作为评判者对解释性回应的可信度如何？对定性研究工作流程的影响
How Trustworthy Are LLM-as-Judge Ratings for Interpretive Responses? Implications for Qualitative Research Workflows
AI总结：
本文探讨了大语言模型作为评判者在解释性回应评估中的可信度，通过比较模型表现与人类评判，指出其在筛选模型时的有效性，但不适合替代人类判断。
摘要：
随着定性研究者日益关注自动化工具在解释性分析中的应用，大型语言模型（LLM）常被直接引入分析流程，缺乏对解释质量的系统评估或模型间的比较。本研究检验LLM作为评判者评估是否与人类对解释质量的判断一致，并能指导模型选择。使用712个半结构化访谈中K-12数学教师的对话片段，生成五种主流推理模型的一句解释性回应。通过AWS Bedrock的LLM-as-judge框架，使用五项指标进行自动化评估，并由训练有素的人类评估者对子集进行独立评分。结果显示，LLM-as-judge评分在模型层面捕捉了人类评分的总体趋势，但在分数幅度上存在显著差异。其中，连贯性在聚合人类评分中表现最强，而忠实度和正确性在片段层面存在系统性偏差，尤其是非字面和细致的解释。安全相关指标与解释质量无关。这些发现表明，LLM-as-judge方法更适合筛选或淘汰表现不佳的模型，而非替代人类判断，为定性研究工作流程中系统比较和选择LLM提供实用指导。
英文摘要：
As qualitative researchers show growing interest in using automated tools to support interpretive analysis, a large language model (LLM) is often introduced into an analytic workflow as is, without systematic evaluation of interpretive quality or comparison across models. This practice leaves model selection largely unexamined despite its potential influence on interpretive outcomes. To address this gap, this study examines whether LLM-as-judge evaluations meaningfully align with human judgments of interpretive quality and can inform model-level decision making. Using 712 conversational excerpts from semi-structured interviews with K-12 mathematics teachers, we generated one-sentence interpretive responses using five widely adopted inference models: Command R+ (Cohere), Gemini 2.5 Pro (Google), GPT-5.1 (OpenAI), Llama 4 Scout-17B Instruct (Meta), and Qwen 3-32B Dense (Alibaba). Automated evaluations were conducted using AWS Bedrock's LLM-as-judge framework across five metrics, and a stratified subset of responses was independently rated by trained human evaluators on interpretive accuracy, nuance preservation, and interpretive coherence. Results show that LLM-as-judge scores capture broad directional trends in human evaluations at the model level but diverge substantially in score magnitude. Among automated metrics, Coherence showed the strongest alignment with aggregated human ratings, whereas Faithfulness and Correctness revealed systematic misalignment at the excerpt level, particularly for non-literal and nuanced interpretations. Safety-related metrics were largely irrelevant to interpretive quality. These findings suggest that LLM-as-judge methods are better suited for screening or eliminating underperforming models than for replacing human judgment, offering practical guidance for systematic comparison and selection of LLMs in qualitative research workflows.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
