近年来，使用自然语言处理技术进行问题难度估计（Question Difficulty Estimation, QDE）的研究激增。基于Transformer的神经网络实现了最先进的性能，主要通过监督学习方法，但也有一项孤立的无监督学习研究。监督方法侧重于预测性能，但需要大量标注数据。另一方面，无监督方法不需要标注数据，但依赖于不同的评估指标，在实践中计算成本也很高。这项工作通过探索用于QDE的主动学习来弥合研究差距，这是一种人在回路的监督方法，旨在最小化标注工作同时匹配最先进模型的性能。主动学习过程在标注子集上迭代训练，仅从人类专家那里获取信息量最大的未标注数据点的标签。此外，我们提出了一种新颖的采集函数PowerVariance，将信息量最大的样本添加到标注集中，这是对分类中流行的PowerBALD函数的回归扩展。我们采用DistilBERT进行QDE，并通过应用Monte Carlo dropout来捕获未标注样本中的认知不确定性，从而识别信息量大的样本。实验表明，使用PowerVariance采集的主动学习在仅标注10%的训练数据后，即可实现接近全监督模型的性能。所提出的方法促进了教育资源的负责任使用，使QDE工具更容易被课程讲师获取，并对个性化支持系统和问答工具等其他应用具有前景。
英文摘要：
In recent years, there has been a surge in research on Question Difficulty Estimation (QDE) using natural language processing techniques. Transformer-based neural networks achieve state-of-the-art performance, primarily through supervised methods but with an isolated study in unsupervised learning. While supervised methods focus on predictive performance, they require abundant labeled data. On the other hand, unsupervised methods do not require labeled data but rely on a different evaluation metric that is also computationally expensive in practice. This work bridges the research gap by exploring active learning for QDE, a supervised human-in-the-loop approach striving to minimize the labeling efforts while matching the performance of state-of-the-art models. The active learning process iteratively trains on a labeled subset, acquiring labels from human experts only for the most informative unlabeled data points. Furthermore, we propose a novel acquisition function PowerVariance to add the most informative samples to the labeled set, a regression extension to the PowerBALD function popular in classification. We employ DistilBERT for QDE and identify informative samples by applying Monte Carlo dropout to capture epistemic uncertainty in unlabeled samples. The experiments demonstrate that active learning with PowerVariance acquisition achieves a performance close to fully supervised models after labeling only 10% of the training data. The proposed methodology promotes the responsible use of educational resources, makes QDE tools more accessible to course instructors, and is promising for other applications such as personalized support systems and question-answering tools.
Incremental Uncertainty-aware Performance Monitoring with Active Labeling Intervention
Siemens AG（西门子公司）；Goethe University Frankfurt（法兰克福歌德大学）；LMU Munich（慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；German Cancer Consortium (DKTK)（德国癌症联盟）
How Does Pseudo-Labeling Affect the Generalization Error of the Semi-Supervised Gibbs Algorithm?
University of Florida（佛罗里达大学）；University College London（伦敦大学学院）；National University of Singapore（新加坡国立大学）；Cornell University（康奈尔大学）；The Alan Turing Institute（艾伦·图灵研究所）
