arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.00374cs.CL
收藏

重新审视自然语言理解中的样本量确定

Revisiting Sample Size Determination in Natural Language Understanding

Meta Inc.(元公司);Saarland University(萨尔大学)

Ernie Chang, Muhammad Hassan Rashid, Pin-Jie Lin, Changsheng Zhao, Vera Demberg, Yangyang Shi, Vikas Chandra

AI总结:

研究NLP中样本量确定问题,提出基于少量训练样本预测最大模型性能的简单方法,仅需10%数据即可在约0.9%平均绝对误差内预测性能,为数据标注提供早期指标。

摘要:

明确知道需要标注多少数据点才能达到一定的模型性能,是减少整体标注预算的极具益处的一步。这涉及主动学习和传统数据标注,对低资源场景尤为有益。然而,在NLP中这仍是一个很大程度上未被充分探索的研究领域。因此,我们探索了多种技术,用于估计达到目标性能值所需的训练样本量。我们推导了一种简单而有效的方法,基于少量训练样本预测最大可实现的模型性能——这在数据标注过程中可作为数据质量和样本量确定的早期指标。我们在四个语言理解任务上进行了消融研究,结果表明所提出的方法仅用10%的数据即可在较小的平均绝对误差(约0.9%)范围内预测模型性能。

英文摘要:

Knowing exactly how many data points need to be labeled to achieve a certain model performance is a hugely beneficial step towards reducing the overall budgets for annotation. It pertains to both active learning and traditional data annotation, and is particularly beneficial for low resource scenarios. Nevertheless, it remains a largely under-explored area of research in NLP. We therefore explored various techniques for estimating the training sample size necessary to achieve a targeted performance value. We derived a simple yet effective approach to predict the maximum achievable model performance based on small amount of training samples - which serves as an early indicator during data annotation for data quality and sample size determination. We performed ablation studies on four language understanding tasks, and showed that the proposed approach allows us to forecast model performance within a small margin of mean absolute error (~ 0.9%) with only 10% data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑