小型模型实现大型语言模型性能:评估具备推理能力的AI用于安全儿童福利研究
Small Models Achieve Large Language Model Performance: Evaluating Reasoning-Enabled AI for Secure Child Welfare Research
AI总结:
本研究展示小型模型通过扩展推理能力在儿童福利研究中实现高准确性,优于大模型,节省资源并提升效率。
摘要:
目的:本研究开发了一个系统性评估框架,用于测试语言模型是否能够准确识别儿童福利记录中的感兴趣构念。目的是评估不同模型大小和架构在四个经过验证的基准测试中对涉及儿童福利家庭中的关键风险因素进行分类的能力:家庭暴力、枪支、一般物质相关问题以及阿片类药物。方法:我们构建了四个基准测试,用于识别儿童福利调查总结中的风险因素:家庭暴力、物质相关问题、枪支和阿片类药物(各500个样本)。我们评估了七个模型大小(0.6B-32B参数)在标准和扩展推理模式下的表现,以及一种专家混合模型变体。Cohen's kappa测量与由人类专家建立的黄金标准分类的一致性。结果:评估揭示了一个关键发现:更大的模型并不更好。一个小型4B参数模型在扩展推理模式下表现最为出色,其性能优于多达八倍大的模型。该模型在所有四个基准类别中都实现了“显著”到“几乎完美”的一致性和准确性。该模型在三个基准测试(物质相关问题、枪支和阿片类药物)上实现了“几乎完美”的一致性(kappa=0.93-0.96),在最复杂的任务(家庭暴力)上实现了“显著”的一致性(kappa=0.74)。具有扩展推理的小型模型在资源效率方面优于最大的模型,同时在准确性上与最大的模型相当。结论:具备推理能力的小型模型实现了历史上需要更大架构才能达到的准确性水平,从而显著节省了时间和计算资源。该评估框架提供了一种基于证据的方法,用于在实际部署前平衡准确性与实际资源限制,以在社会工作研究中应用。
英文摘要:
Objective: This study develops a systematic benchmarking framework for testing whether language models can accurately identify constructs of interest in child welfare records. The objective is to assess how different model sizes and architectures perform on four validated benchmarks for classifying critical risk factors among child welfare-involved families: domestic violence, firearms, substance-related problems generally, and opioids specifically. Method: We constructed four benchmarks for identifying risk factors in child welfare investigation summaries: domestic violence, substance-related problems, firearms, and opioids (n=500 each). We evaluated seven model sizes (0.6B-32B parameters) in standard and extended reasoning modes, plus a mixture-of-experts variant. Cohen's kappa measured agreement with gold standard classifications established by human experts. Results: The benchmarking revealed a critical finding: bigger models are not better. A small 4B parameter model with extended reasoning proved most effective, outperforming models up to eight times larger. It consistently achieved "substantial" to "almost perfect" agreement across all four benchmark categories. This model achieved "almost perfect" agreement (\k{appa} = 0.93-0.96) on three benchmarks (substance-related problems, firearms, and opioids) and "substantial" agreement (\k{appa} = 0.74) on the most complex task (domestic violence). Small models with extended reasoning rivaled the largest models while being more resource-efficient. Conclusions: Small reasoning-enabled models achieve accuracy levels historically requiring larger architectures, enabling significant time and computational efficiencies. The benchmarking framework provides a method for evidence-based model selection to balance accuracy with practical resource constraints before operational deployment in social work research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。