量化基于大语言模型的公共话语立场分析中的不稳定来源

中文摘要 AI 辅助

计算社会科学越来越依赖自动化预处理管道将原始媒体转换为可分析文本。当这些管道对相同输入产生不同输出时，会出现两种不同的不稳定来源：预处理管道本身和下游测量工具。通过对来自五个领域的41位公众人物的256次YouTube采访进行研究，比较了两种说话人日记化管道和两种测量方法。发现预处理管道敏感性集中在视频样本有限的说话人；跨方法分歧更大且更系统；总体效价比例高度稳定。贡献是一个诊断框架，可将管道效应与测量效应分开，研究人员应验证结论对两种变化来源的稳健性，尤其要注意测量方法的选择。

英文摘要

Computational social science increasingly relies on automated preprocessing pipelines -- speaker diarization, ASR transcript cleaning, sentence segmentation -- to convert raw media into analyzable text. When these pipelines produce different outputs from the same input, two distinct sources of instability can arise: the preprocessing pipeline itself (diarization method, segmentation rules) and the downstream measurement instrument (LLM annotation vs.\ keyword lexicon). Using 256 YouTube interviews across 41 public figures from five domains, we compare two speaker-diarization pipelines and two measurement methods, all targeting the coupling between affective valence and epistemic modality. We find that (1) preprocessing pipeline sensitivity is concentrated in speakers with limited video samples (N $\leq 5$); for the four best-sampled speakers (N $\geq 16$), the mean absolute pipeline-induced change in $r(\text{neg}, \text{emph})$ is only $0.13$; (2) cross-method disagreement is larger and more systematic -- the LLM and keyword-lexicon methods assign opposite coupling directions to several well-sampled speakers, even within the same preprocessing pipeline; and (3) aggregate valence proportions are highly stable ($|Δp(\text{neg})| < 6$pp) regardless of pipeline or method, masking both sources of instability. The contribution is a diagnostic framework that separates pipeline effects from measurement effects: researchers studying cross-dimensional relationships in interview data should verify that their conclusions are robust to both sources of variation, with particular attention to measurement method choice.

发表机构

• Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）

机构由 AI 辅助整理，请以论文原文为准。