从叙事到数字:利用语言模型对口头尸检叙事的预测进行有效推断
From Narratives to Numbers: Valid Inference Using Language Model Predictions from Verbal Autopsy Narratives
AI总结:
本文提出multiPPI++方法,将预测驱动推断扩展到多项分类,以校正NLP从口头尸检文本预测死因后的统计推断,并在VA数据上验证其跨模型可迁移性。
摘要:
在大多数死亡发生于医疗系统之外的环境中,口头尸检(VA)是监测死因(COD)趋势的常用工具。VA是对在世照护者或亲属进行的访谈,用于预测死者的COD。将VA转化为研究人员和政策制定者可采取行动的洞见需要两个步骤:(i) 利用VA访谈预测可能的COD;(ii) 利用预测的COD进行推断(例如,使用死亡样本按人口统计学因素对死因构成进行建模)。在本文中,我们开发了一种利用最先进NLP技术从自由格式文本预测的结果(在我们的案例中为COD)进行有效推断的方法。该方法我们称为multiPPI++,将“预测驱动推断”方面的最新工作扩展到多项分类。我们利用一套NLP技术进行COD预测,并通过对VA数据的实证分析,证明了我们的方法在处理可迁移性问题方面的有效性。无论预测由哪种NLP模型产生,也无论它们是由GPT-4-32k等更准确的预测器还是KNN等不太准确的预测器产生,multiPPI++都能恢复真实估计值。我们的研究结果证明了推断校正在公共卫生决策中的实际重要性,并表明如果推断任务是最终目标,那么无论采用何种NLP算法,拥有少量与情境相关的高质量标注数据都至关重要。
英文摘要:
In settings where most deaths occur outside the healthcare system, verbal autopsies (VAs) are a common tool to monitor trends in causes of death (COD). VAs are interviews with a surviving caregiver or relative that are used to predict the decedent's COD. Turning VAs into actionable insights for researchers and policymakers requires two steps (i) predicting likely COD using the VA interview and (ii) performing inference with predicted CODs (e.g. modeling the breakdown of causes by demographic factors using a sample of deaths). In this paper, we develop a method for valid inference using outcomes (in our case COD) predicted from free-form text using state-of-the-art NLP techniques. This method, which we call multiPPI++, extends recent work in "prediction-powered inference" to multinomial classification. We leverage a suite of NLP techniques for COD prediction and, through empirical analysis of VA data, demonstrate the effectiveness of our approach in handling transportability issues. multiPPI++ recovers ground truth estimates, regardless of which NLP model produced predictions and regardless of whether they were produced by a more accurate predictor like GPT-4-32k or a less accurate predictor like KNN. Our findings demonstrate the practical importance of inference correction for public health decision-making and suggests that if inference tasks are the end goal, having a small amount of contextually relevant, high quality labeled data is essential regardless of the NLP algorithm.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。