从声音到效度：利用大语言模型（LLMs）开展政策利益相关者访谈的文本分析
From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviews
AI总结：
本研究针对政策访谈人工编码耗时耗力的问题，提出人机结合的混合方法，结合人类领域知识与GPT-4开展K-12教育政策访谈的主题与情感分析，显著提升了分析效率、效度与可解释性。
摘要：
及时获取利益相关者对当前政策的多元经验与意见，对政策制定者识别资源分配的优势与缺口、从而支撑有效的政策设计与落地至关重要。然而，即便对规模适中的利益相关者访谈文本或开放式调查回复进行人工编码，往往也耗费大量人力与时间。本研究探索将GPT-4等大语言模型（LLMs）与人类专业知识相结合，以优化美国某州K-12教育政策相关的利益相关者访谈文本分析效果。研究采用混合研究方法：人类专家结合领域知识与无监督主题建模结果，制定编码本与编码流程；随后设计提示词引导GPT-4开展分析，并迭代评估不同提示词的表现。这种人机结合的方法实现了细致的主题与情感分析。结果显示，GPT-4的主题编码在具体主题层面与人工编码的一致性达77.89%，扩展至更宽泛主题时一致性提升至96.02%，较传统自然语言处理（NLP）方法高出25%以上。此外，相较于基于词典的方法，GPT-4的结果与专家情感分析的匹配度更高。量化指标与质性评审的结果共同凸显了人类领域专业知识与自动化分析的互补作用——LLM能提供新视角与编码一致性。这种人机交互方法提升了教育政策研究的效率、效度与可解释性。
英文摘要：
Obtaining stakeholders' diverse experiences and opinions about current policy in a timely manner is crucial for policymakers to identify strengths and gaps in resource allocation, thereby supporting effective policy design and implementation. However, manually coding even moderately sized interview texts or open-ended survey responses from stakeholders can often be labor-intensive and time-consuming. This study explores the integration of Large Language Models (LLMs)--like GPT-4--with human expertise to enhance text analysis of stakeholder interviews regarding K-12 education policy within one U.S. state. Employing a mixed-methods approach, human experts developed a codebook and coding processes as informed by domain knowledge and unsupervised topic modeling results. They then designed prompts to guide GPT-4 analysis and iteratively evaluate different prompts' performances. This combined human-computer method enabled nuanced thematic and sentiment analysis. Results reveal that while GPT-4 thematic coding aligned with human coding by 77.89% at specific themes, expanding to broader themes increased congruence to 96.02%, surpassing traditional Natural Language Processing (NLP) methods by over 25%. Additionally, GPT-4 is more closely matched to expert sentiment analysis than lexicon-based methods. Findings from quantitative measures and qualitative reviews underscore the complementary roles of human domain expertise and automated analysis as LLMs offer new perspectives and coding consistency. The human-computer interactive approach enhances efficiency, validity, and interpretability of educational policy research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
