平台是否是测量的一部分?面向机器标注的政策情感跨平台等价性的协议与模拟研究
Is the Platform Part of the Measurement? A Protocol and Simulation Study for Cross-Platform Equivalence of Machine-Labelled Policy Sentiment
AI总结:
本研究将心理测量等价性工具包适配机器标注政策情感,提出含锚定事件等的六步协议,经模拟与案例研究表明,该协议可避免跨平台合并得分时的情感符号错误,为跨平台情感测量提供方法支撑。
摘要:
跨社交媒体平台比较机器标注的情感如今已成为常规操作:不同平台间分类器得分的差异被解读为公众意见的差异。这类比较基于一个假设,即分类器在每个平台上测量的是同一构念,且使用相同的量表。一些研究会逐一检验分类器在各平台上的有效性,但据我们所知,尚无研究将平台建模为分类器测量的一个维度。我们将心理测量等价性工具包(概化理论、测量不变性与差异项目功能、分数链接)适配到机器标注的政策情感研究中,将平台视为一种调查模式,并制定了一个六步协议,包含预先固定的锚定事件、锚定密度审计以及双分类器收敛控制。一项具有已知真实值的模拟研究展示了该假设在一个预先固定的生成配置中会造成何种代价:在500次重复实验中,由于各平台通过不同函数进行测量,直接采用平台得分的合并指数在近四分之一的天数中报告了错误的总体情感符号,而使用各平台已知测量函数调整后的指数平均每六十天中出错不足一天。对第一作者硕士论文中的工具(一个在来自Twitter/X、Reddit、YouTube和Quora的255446条英国经济政策评论上微调的BERT分类器)的案例研究,为该协议的设计提供了动机,并说明了该协议所解决的工具依赖性问题。根据预先指定的设计进行新采集的完整四平台应用,是指定的后续研究。该协议允许研究者检验在给定语料库中合并平台是否合理,或在取任何平均值之前是否必须对平台得分进行链接。
英文摘要:
Comparing machine-labelled sentiment across social media platforms is now routine: differences in classifier scores between platforms are read as differences in public opinion. These comparisons rest on an assumption that the classifier measures the same construct, on the same scale, on every platform. Some studies check a classifier's validity platform by platform, but to our knowledge none has modelled the platform as a facet of the classifier's measurement. We adapt the psychometric equivalence toolkit (generalizability theory, measurement invariance and differential item functioning, and score linking) to machine-labelled policy sentiment, treating the platform as a survey mode, and specify a six-step protocol with anchor events fixed in advance, an anchor-density audit, and a two-classifier convergence control. A simulation with known ground truth illustrates what the assumption can cost in one generating configuration fixed in advance: across 500 replications, because the platforms measure through different functions, a pooled index that takes platform scores at face value reports the wrong sign of aggregate sentiment on nearly a quarter of days, whereas an index adjusted with each platform's known measurement function does so on under one day in sixty on average. A case study of the instrument from the first author's MSc dissertation, a fine-tuned BERT classifier over 255,446 UK economic-policy comments from Twitter/X, Reddit, YouTube, and Quora, motivates the design and illustrates the instrument-dependence the protocol addresses. The full four-platform application, under fresh collection with the pre-specified design, is the designated follow-up study. The protocol lets a researcher test whether pooling platforms is defensible in a given corpus, or whether platform scores must be linked before any average is taken.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。