文本嵌入的结构效度评价及其在问卷调查中的应用
Evaluating the Construct Validity of Text Embeddings with Application to Survey Questions
摘要：
来自自然语言处理的文本嵌入模型可以将文本数据（例如单词、句子、文档）映射到假定有意义的数字表示（也称为文本嵌入）。虽然这些模型越来越多地应用于社会科学研究，但一个重要的问题往往没有得到解决：这些嵌入在多大程度上是与社会科学研究相关的结构的有效表示。因此，我们建议使用经典的结构效度框架来评估文本嵌入的有效性。我们展示了这个框架如何适应文本嵌入的不透明和高维性质，并应用于调查问题。在我们的结构有效性分析中，我们包括几种流行的文本嵌入方法（例如fastText，GloVe，BERT，Sentence-BERT，Universal Sentence Encoder）。在某些情况下，我们发现收敛和判别有效性的证据。我们还表明，嵌入可以用来预测受访者的答案，全新的调查问题。此外，基于BERT的嵌入技术和通用语句编码器提供了比其他更有效的调查问题表示。因此，我们的研究结果突出了在社会科学研究中部署文本嵌入之前检查其结构效度的必要性。
英文摘要：
Text embedding models from Natural Language Processing can map text data (e.g. words, sentences, documents) to supposedly meaningful numerical representations (a.k.a. text embeddings). While such models are increasingly applied in social science research, one important issue is often not addressed: the extent to which these embeddings are valid representations of constructs relevant for social science research. We therefore propose the use of the classic construct validity framework to evaluate the validity of text embeddings. We show how this framework can be adapted to the opaque and high-dimensional nature of text embeddings, with application to survey questions. We include several popular text embedding methods (e.g. fastText, GloVe, BERT, Sentence-BERT, Universal Sentence Encoder) in our construct validity analyses. We find evidence of convergent and discriminant validity in some cases. We also show that embeddings can be used to predict respondent's answers to completely new survey questions. Furthermore, BERT-based embedding techniques and the Universal Sentence Encoder provide more valid representations of survey questions than do others. Our results thus highlight the necessity to examine the construct validity of text embeddings before deploying them in social science research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
