关于归纳主题饱和度作为衡量使用LLM进行归纳主题分析有效性潜在指标的思考
Reflections on Inductive Thematic Saturation as a potential metric for measuring the validity of an inductive Thematic Analysis with LLMs
AI总结:
本文探讨使用LLM进行主题分析,提出利用初始主题饱和度(ITS)作为评估初始编码交易有效性的指标,并通过累计与唯一编码斜率之比进行综合测量。
摘要:
本文提出了一系列关于饱和度以及使用大型语言模型(LLMs)进行主题分析(TA)的思考。本文建议,初始主题饱和度(ITS)可作为一个指标,用于评估使用LLM进行TA的部分交易有效性,重点关注初始编码。本文展示了两个不同规模数据集的初始编码,并反思了LLM在编码过程中如何达到某种形式的分析饱和度。本工作提出的程序促成了两个编码手册的创建,一个包含累计初始编码总数,另一个包含唯一编码总数。本文提出了一个指标,通过使用累计编码与唯一编码斜率之比的简单数学计算来综合测量ITS。本文为探索如何使用LLM进行定性分析的早期工作体系做出了贡献。
英文摘要:
This paper presents a set of reflections on saturation and the use of Large Language Models (LLMs) for performing Thematic Analysis (TA). The paper suggests that initial thematic saturation (ITS) could be used as a metric to assess part of the transactional validity of TA with LLM, focusing on the initial coding. The paper presents the initial coding of two datasets of different sizes, and it reflects on how the LLM reaches some form of analytical saturation during the coding. The procedure proposed in this work leads to the creation of two codebooks, one comprising the total cumulative initial codes and the other the total unique codes. The paper proposes a metric to synthetically measure ITS using a simple mathematical calculation employing the ratio between slopes of cumulative codes and unique codes. The paper contributes to the initial body of work exploring how to perform qualitative analysis with LLMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。