利用大型语言模型和合成数据监测研究论文中的数据集提及
Large Language Models and Synthetic Data for Monitoring Dataset Mentions in Research Papers
AI总结:
本文提出结合 LLM、弱监督合成数据与两阶段微调的数据集提及自动监测框架,并以 ModernBERT 过滤提升效率,实验显示其抽取准确率优于 NuExtract-v1.5 和 GLiNER-large-v2.1。
摘要:
追踪数据在研究论文中被提及和使用的方式,可为提升数据可发现性、质量和生产提供关键洞见。然而,在海量学术文献中人工识别并分类数据集提及既耗费资源,又无法扩展。本文提出一种机器学习框架,通过利用大型语言模型(LLM)、合成数据和两阶段微调流程,实现跨研究领域的数据集提及检测自动化。我们采用从研究论文中进行零样本抽取、用 LLM-as-a-Judge 进行质量评估,并用推理智能体进行精炼,以生成弱监督合成数据集。Phi-3.5-mini instruct 模型先在该数据集上进行预微调,随后在人工标注子集上进行微调。推理时,基于 ModernBERT 的分类器高效过滤数据集提及,在保持高召回率的同时降低计算开销。在一个留出的人工标注样本上评估时,我们的微调模型在数据集抽取准确率上优于 NuExtract-v1.5 和 GLiNER-large-v2.1。结果表明,LLM 生成的合成数据能够有效缓解训练数据稀缺问题,提升低资源场景下的泛化能力。该框架为可扩展地监测数据集使用、增强透明度提供了路径,并支持研究人员、资助方和政策制定者识别数据缺口、加强数据可及性,从而作出明智决策。
英文摘要:
Tracking how data is mentioned and used in research papers provides critical insights for improving data discoverability, quality, and production. However, manually identifying and classifying dataset mentions across vast academic literature is resource-intensive and not scalable. This paper presents a machine learning framework that automates dataset mention detection across research domains by leveraging large language models (LLMs), synthetic data, and a two-stage fine-tuning process. We employ zero-shot extraction from research papers, an LLM-as-a-Judge for quality assessment, and a reasoning agent for refinement to generate a weakly supervised synthetic dataset. The Phi-3.5-mini instruct model is pre-fine-tuned on this dataset, followed by fine-tuning on a manually annotated subset. At inference, a ModernBERT-based classifier efficiently filters dataset mentions, reducing computational overhead while maintaining high recall. Evaluated on a held-out manually annotated sample, our fine-tuned model outperforms NuExtract-v1.5 and GLiNER-large-v2.1 in dataset extraction accuracy. Our results highlight how LLM-generated synthetic data can effectively address training data scarcity, improving generalization in low-resource settings. This framework offers a pathway toward scalable monitoring of dataset usage, enhancing transparency, and supporting researchers, funders, and policymakers in identifying data gaps and strengthening data accessibility for informed decision-making.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。